llm

package
v0.0.0-...-605214e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 30, 2026 License: GPL-3.0 Imports: 16 Imported by: 0

Documentation

Overview

Package llm: provider_chain assembles the runtime Provider decorator stack.

NewProviderChain wraps a primary provider, innermost to outermost, with cache, fallback, retry, throttle, per-call timeout and budget layers. The throttle layer is process-wide when a throttle key is set (see WithThrottleKey): every chain built for the same provider name shares one semaphore so LLM_THROTTLE_CONCURRENCY bounds concurrent calls across all runs, not per chain.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrProviderNotFound indicates that no provider has been registered for a name.
	ErrProviderNotFound = errors.New("llm provider not found")
	// ErrModelTierNotConfigured indicates that a provider was found but has no model for the requested tier.
	ErrModelTierNotConfigured = errors.New("llm model tier not configured")
)
View Source 
var ErrBudgetExhausted = errors.New("llm: daily budget exhausted")

ErrBudgetExhausted is returned when daily request or token budget is spent. RetryProvider treats this as non-retryable (unknown error type, no statusCoder).

Functions

func PromptBytes 

func PromptBytes(messages []Message) int

PromptBytes returns the UTF-8 byte size of all message contents.

func ValidatePromptBytes 

func ValidatePromptBytes(messages []Message, maxBytes int) error

ValidatePromptBytes rejects prompts that exceed maxBytes.

func WithCacheStatsCollector 

func WithCacheStatsCollector(ctx context.Context, collector *CacheStatsCollector) context.Context

WithCacheStatsCollector attaches collector to ctx.

Types

type AnthropicProviderConfig 

type AnthropicProviderConfig struct {
	APIKey  string
	BaseURL string
	Model   string
}

AnthropicProviderConfig is the config shape required by an Anthropic factory.

type Budget 

type Budget struct {
	// contains filtered or unexported fields
}

Budget tracks daily request and token usage with thread-safe counters. Zero limits mean unlimited. Call Reset() to clear counters (e.g. at UTC midnight).

func BuildLLMBudget 

func BuildLLMBudget(cfg config.LLMConfig) *Budget

BuildLLMBudget converts config budget fields into an LLM budget guard.

func NewBudget 

func NewBudget(maxRequestsPerDay, maxTokensPerDay int) *Budget

NewBudget creates a Budget with the given daily limits. Zero values for either limit mean that dimension is unlimited.

func (*Budget) Allow 

func (b *Budget) Allow() bool

Allow returns true if a new request is within budget. It auto-resets counters when UTC midnight has passed.

func (*Budget) Record 

func (b *Budget) Record(promptTokens, completionTokens int)

Record tracks token usage for a completed request.

func (*Budget) Reset 

func (b *Budget) Reset()

Reset clears all counters and sets the next reset time.

func (*Budget) Stats 

func (b *Budget) Stats() BudgetStats

Stats returns current budget usage (for observability).

type BudgetGuardProvider 

type BudgetGuardProvider struct {
	// contains filtered or unexported fields
}

BudgetGuardProvider wraps a Provider, rejecting calls when budget is exhausted. On success it records token usage back to the budget.

func NewBudgetGuardProvider 

func NewBudgetGuardProvider(inner Provider, budget *Budget) *BudgetGuardProvider

NewBudgetGuardProvider wraps inner with budget enforcement.

func (*BudgetGuardProvider) Complete 

func (b *BudgetGuardProvider) Complete(ctx context.Context, req CompletionRequest) (*CompletionResponse, error)

Complete checks budget before delegating. Records usage after success.

func (*BudgetGuardProvider) WithBudgetMetrics 

func (b *BudgetGuardProvider) WithBudgetMetrics(m BudgetMetrics) *BudgetGuardProvider

WithBudgetMetrics attaches optional budget metrics.

type BudgetMetrics 

type BudgetMetrics interface {
	RecordLLMBudgetExhausted()
}

BudgetMetrics captures budget exhaustion events for observability.

type BudgetStats 

type BudgetStats struct {
	Requests          int
	MaxRequestsPerDay int
	PromptTokens      int
	CompletionTokens  int
	TotalTokens       int
	MaxTokensPerDay   int
	ResetAt           time.Time
}

BudgetStats holds a snapshot of budget usage.

type CacheMetrics 

type CacheMetrics interface {
	RecordLLMCacheHit()
	RecordLLMCacheMiss()
}

CacheMetrics records aggregate cache hits and misses.

type CacheProvider 

type CacheProvider struct {
	// contains filtered or unexported fields
}

CacheProvider wraps a Provider with response caching.

func NewCacheProvider 

func NewCacheProvider(provider Provider, cache ResponseCache, version string) (*CacheProvider, error)

NewCacheProvider wraps provider with a response cache. An empty version uses the default cache version.

func (*CacheProvider) Complete 

func (c *CacheProvider) Complete(ctx context.Context, request CompletionRequest) (*CompletionResponse, error)

Complete returns a cached response when available, otherwise it delegates to the wrapped provider and stores the result.

func (*CacheProvider) WithCacheMetrics 

func (c *CacheProvider) WithCacheMetrics(m CacheMetrics) *CacheProvider

WithCacheMetrics attaches optional cache metrics.

type CacheStats 

type CacheStats struct {
	Hits     int     `json:"hits"`
	Misses   int     `json:"misses"`
	Requests int     `json:"requests"`
	HitRate  float64 `json:"hit_rate"`
	MissRate float64 `json:"miss_rate"`
}

CacheStats summarizes cache behavior for a single run.

type CacheStatsCollector 

type CacheStatsCollector struct {
	// contains filtered or unexported fields
}

CacheStatsCollector records per-context cache hits and misses.

func CacheStatsCollectorFromContext 

func CacheStatsCollectorFromContext(ctx context.Context) *CacheStatsCollector

CacheStatsCollectorFromContext returns the collector stored on ctx, if any.

func NewCacheStatsCollector 

func NewCacheStatsCollector() *CacheStatsCollector

NewCacheStatsCollector returns an empty stats collector.

func (*CacheStatsCollector) Snapshot 

func (c *CacheStatsCollector) Snapshot() CacheStats

Snapshot returns the current hit/miss totals and rates.

type CachedProvider 

type CachedProvider = CacheProvider

CachedProvider is a compatibility alias for CacheProvider.

func NewCachedProvider 

func NewCachedProvider(provider Provider, cache ResponseCache) *CachedProvider

NewCachedProvider wraps provider with a response cache using the default cache version. It returns nil when provider or cache is invalid.

type ChainOption 

type ChainOption func(*chainConfig)

ChainOption configures the provider chain built by NewProviderChain.

func WithBudget 

func WithBudget(b *Budget) ChainOption

WithBudget attaches a budget guard that rejects calls when daily limits are hit.

func WithCache 

func WithCache(cache ResponseCache) ChainOption

WithCache enables response caching using the given cache store.

func WithCallTimeout 

func WithCallTimeout(d time.Duration) ChainOption

WithCallTimeout wraps each Complete call with a per-call context timeout. Zero or negative values disable per-call timeout.

func WithChainBudgetMetrics 

func WithChainBudgetMetrics(m BudgetMetrics) ChainOption

WithChainBudgetMetrics attaches metrics to the budget guard layer.

func WithChainCacheMetrics 

func WithChainCacheMetrics(m CacheMetrics) ChainOption

WithChainCacheMetrics attaches metrics to the cache layer.

func WithChainFallbackMetrics 

func WithChainFallbackMetrics(m FallbackMetrics) ChainOption

WithChainFallbackMetrics attaches metrics to the fallback layer.

func WithChainRetryMetrics 

func WithChainRetryMetrics(m RetryMetrics) ChainOption

WithChainRetryMetrics attaches metrics to the retry layer.

func WithFallback 

func WithFallback(p Provider) ChainOption

WithFallback adds a secondary provider tried when primary fails.

func WithRetry 

func WithRetry(maxAttempts int) ChainOption

WithRetry sets the max retry attempts (including initial call). Values < 1 are ignored.

func WithRetryBaseDelay 

func WithRetryBaseDelay(d time.Duration) ChainOption

WithRetryBaseDelay sets the base delay for exponential backoff.

func WithThrottle 

func WithThrottle(n int) ChainOption

WithThrottle sets the max concurrent calls. Values < 1 are clamped to 1.

func WithThrottleKey 

func WithThrottleKey(key string) ChainOption

WithThrottleKey makes the throttle layer share the process-wide semaphore registered under key (typically the provider name). Chains built without a key use a private semaphore.

type CompletionRequest 

type CompletionRequest struct {
	Model          string          `json:"model,omitempty"`
	Messages       []Message       `json:"messages"`
	Temperature    float64         `json:"temperature,omitempty"`
	MaxTokens      int             `json:"max_tokens,omitempty"`
	ResponseFormat *ResponseFormat `json:"response_format,omitempty"`
}

CompletionRequest captures the provider-agnostic input required for a chat completion.

type CompletionResponse 

type CompletionResponse struct {
	Content      string          `json:"content"`
	Usage        CompletionUsage `json:"usage"`
	Model        string          `json:"model,omitempty"`
	LatencyMS    int             `json:"latency_ms,omitempty"`
	CostUSD      float64         `json:"cost_usd,omitempty"`
	UsedFallback bool            `json:"used_fallback,omitempty"`
	TimedOut     bool            `json:"timed_out,omitempty"`
}

CompletionResponse captures the provider-agnostic result of a chat completion.

type CompletionUsage 

type CompletionUsage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
}

CompletionUsage tracks token counts returned by the provider.

type Composer 

type Composer struct {
	// contains filtered or unexported fields
}

Composer owns runtime LLM provider assembly.

func NewComposer 

func NewComposer(factories RuntimeProviderFactories) Composer

NewComposer constructs a runtime composer from provider factories.

func (Composer) BuildProvider 

func (c Composer) BuildProvider(cfg config.LLMConfig, appMetrics any, logger *slog.Logger, budget *Budget) Provider

BuildProvider builds the configured primary provider and wraps it with the runtime chain.

func (Composer) BuildProviderForSelection 

func (c Composer) BuildProviderForSelection(cfg config.LLMConfig, providerName, model string, logger *slog.Logger) (Provider, error)

BuildProviderForSelection resolves and constructs a provider for a requested provider name.

func (Composer) WrapProviderChain 

func (c Composer) WrapProviderChain(primary Provider, cfg config.LLMConfig, appMetrics any, logger *slog.Logger, budget *Budget) Provider

WrapProviderChain wraps an existing provider with runtime chain layers.

type FallbackMetrics 

type FallbackMetrics interface {
	RecordLLMFallback(reason string)
}

FallbackMetrics captures fallback events for observability.

type FallbackProvider 

type FallbackProvider struct {
	// contains filtered or unexported fields
}

FallbackProvider wraps a primary and secondary Provider. If the primary provider fails, the request is retried against the secondary provider. Fallback events are logged for observability.

Context cancellation errors are returned immediately. DeadlineExceeded errors are retried against the secondary provider using a fresh context.

func NewFallbackProvider 

func NewFallbackProvider(primary, secondary Provider, logger *slog.Logger) (*FallbackProvider, error)

NewFallbackProvider constructs a FallbackProvider that tries primary first and falls back to secondary on non-context errors. If logger is nil, slog.Default() is used.

func (*FallbackProvider) Complete 

func (f *FallbackProvider) Complete(ctx context.Context, request CompletionRequest) (*CompletionResponse, error)

Complete tries the primary provider first. On failure it logs the event and attempts the secondary provider. Context.Canceled is returned immediately without fallback. Context.DeadlineExceeded is retried against the secondary provider with a fresh context. If both providers fail the secondary error is returned.

func (*FallbackProvider) WithMetrics 

func (f *FallbackProvider) WithMetrics(metrics FallbackMetrics) *FallbackProvider

WithMetrics wires an optional metrics recorder into the provider.

type GoogleProviderConfig 

type GoogleProviderConfig struct {
	APIKey  string
	BaseURL string
	Model   string
}

GoogleProviderConfig is the config shape required by a Google factory.

type MemoryResponseCache 

type MemoryResponseCache struct {
	// contains filtered or unexported fields
}

MemoryResponseCache is a capacity-bounded in-memory ResponseCache implementation. When the cache is full an arbitrary entry is evicted before inserting the new one, bounding memory consumption to at most maxItems entries.

func NewMemoryResponseCache 

func NewMemoryResponseCache() *MemoryResponseCache

NewMemoryResponseCache returns an empty in-memory response cache bounded to defaultMemoryCacheMaxItems entries.

func NewMemoryResponseCacheWithLimit 

func NewMemoryResponseCacheWithLimit(maxItems int) *MemoryResponseCache

NewMemoryResponseCacheWithLimit returns an empty in-memory response cache bounded to maxItems entries. If maxItems is <= 0 the default limit is used.

func (*MemoryResponseCache) Get 

func (c *MemoryResponseCache) Get(key string) (*CompletionResponse, bool)

Get returns a cloned cached response for the given key.

func (*MemoryResponseCache) Set 

func (c *MemoryResponseCache) Set(key string, response *CompletionResponse)

Set stores a cloned response for the given key. If the cache is at capacity an arbitrary entry is evicted first to keep the total entry count bounded.

type Message 

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

Message represents a single chat message in a completion request.

type ModelTier 

type ModelTier string

ModelTier identifies the level of reasoning required for a request. 

const (
	// ModelTierDeepThink is intended for higher-quality, more expensive reasoning tasks.
	ModelTierDeepThink ModelTier = "deep_think"
	// ModelTierQuickThink is intended for faster, lower-cost reasoning tasks.
	ModelTierQuickThink ModelTier = "quick_think"
)

func (ModelTier) String 

func (t ModelTier) String() string

String returns the string representation of a ModelTier.

type OllamaProviderConfig 

type OllamaProviderConfig struct {
	BaseURL string
	APIKey  string
	Model   string
}

OllamaProviderConfig is the config shape required by an Ollama factory.

type OpenAIProviderConfig 

type OpenAIProviderConfig struct {
	APIKey  string
	BaseURL string
	Model   string
}

OpenAIProviderConfig is the config shape required by an OpenAI-compatible factory.

type OpenCodeProviderConfig 

type OpenCodeProviderConfig struct {
	BaseURL         string
	Username        string
	Password        string
	Model           string
	UseRequestModel bool
}

OpenCodeProviderConfig is the config shape required by an OpenCode factory.

type PromptTooLargeError 

type PromptTooLargeError struct {
	ActualBytes int
	MaxBytes    int
}

PromptTooLargeError reports a prompt body that exceeds a byte ceiling.

func (*PromptTooLargeError) Error 

func (e *PromptTooLargeError) Error() string

type Provider 

type Provider interface {
	Complete(ctx context.Context, request CompletionRequest) (*CompletionResponse, error)
}

Provider defines the minimal interface required to execute an LLM completion.

func NewProviderChain 

func NewProviderChain(primary Provider, logger *slog.Logger, opts ...ChainOption) Provider

NewProviderChain composes a resilient provider from existing primitives.

Chain order (outermost → innermost): 

budget guard → timeout → throttle → retry → fallback → cache → raw provider

Each layer is optional; only layers whose options are provided are added. The resulting Provider delegates to the composed chain.

type ProviderFunc 

type ProviderFunc func(ctx context.Context, request CompletionRequest) (*CompletionResponse, error)

ProviderFunc adapts a plain function to the Provider interface.

func (ProviderFunc) Complete 

func (f ProviderFunc) Complete(ctx context.Context, request CompletionRequest) (*CompletionResponse, error)

Complete delegates to the wrapped function.

type ProviderRegistration 

type ProviderRegistration struct {
	Provider Provider
	Models   map[ModelTier]string
}

ProviderRegistration stores a provider along with the models assigned to each tier.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry stores providers by name and their tier-specific model mappings.

func NewRegistry 

func NewRegistry() *Registry

NewRegistry constructs an empty provider registry.

func (*Registry) Get 

func (r *Registry) Get(name string) (ProviderRegistration, bool)

Get returns the registered provider entry for a name.

func (*Registry) Register 

func (r *Registry) Register(name string, provider Provider, models map[ModelTier]string) error

Register stores a provider registration under the provided name.

func (*Registry) Resolve 

func (r *Registry) Resolve(name string, tier ModelTier) (Provider, string, error)

Resolve returns the provider and model configured for the requested provider name and tier.

type ResponseCache 

type ResponseCache interface {
	Get(key string) (*CompletionResponse, bool)
	Set(key string, response *CompletionResponse)
}

ResponseCache stores completion responses by cache key.

type ResponseFormat 

type ResponseFormat struct {
	Type   ResponseFormatType `json:"type"`
	Schema json.RawMessage    `json:"schema,omitempty"`
}

ResponseFormat describes the expected response shape.

Schema is optional and allows future providers to pass through a JSON schema for structured output modes supported by OpenAI, Anthropic, or Google APIs.

type ResponseFormatType 

type ResponseFormatType string

ResponseFormatType identifies the requested output format for a completion. 

const (
	// ResponseFormatText requests plain text output.
	ResponseFormatText ResponseFormatType = "text"
	// ResponseFormatJSONObject requests structured JSON object output.
	ResponseFormatJSONObject ResponseFormatType = "json_object"
)

type RetryMetrics 

type RetryMetrics interface {
	RecordLLMRetry()
}

RetryMetrics captures retry events for observability.

type RetryOption 

type RetryOption func(*RetryProvider)

RetryOption configures a RetryProvider.

func WithBaseDelay 

func WithBaseDelay(d time.Duration) RetryOption

WithBaseDelay sets the base delay for exponential backoff.

func WithMaxAttempts 

func WithMaxAttempts(n int) RetryOption

WithMaxAttempts sets the maximum number of attempts (including the first).

type RetryProvider 

type RetryProvider struct {
	// contains filtered or unexported fields
}

RetryProvider wraps a Provider with exponential-backoff retry logic. It retries on transient errors (rate limits, server errors, timeouts) and does not retry on client errors (bad request, auth failures).

func NewRetryProvider 

func NewRetryProvider(provider Provider, logger *slog.Logger, opts ...RetryOption) *RetryProvider

NewRetryProvider wraps provider with retry logic using exponential backoff. If logger is nil, slog.Default() is used.

func (*RetryProvider) Complete 

func (r *RetryProvider) Complete(ctx context.Context, request CompletionRequest) (*CompletionResponse, error)

Complete executes the completion request with retry logic. Token usage is aggregated across all attempts (including failed ones that return partial usage).

func (*RetryProvider) SetTimerFn 

func (r *RetryProvider) SetTimerFn(fn func(time.Duration) (<-chan time.Time, func() bool))

SetTimerFn overrides the function used to create backoff timers between retries. This is intended for testing only. The function should return a channel that fires after the duration and a stop function to release resources.

func (*RetryProvider) WithRetryMetrics 

func (r *RetryProvider) WithRetryMetrics(m RetryMetrics) *RetryProvider

WithRetryMetrics attaches optional retry metrics.

type RuntimeProviderFactories 

type RuntimeProviderFactories struct {
	OpenAI     func(OpenAIProviderConfig) (Provider, error)
	Anthropic  func(AnthropicProviderConfig) (Provider, error)
	Google     func(GoogleProviderConfig) (Provider, error)
	OpenRouter func(OpenAIProviderConfig) (Provider, error)
	XAI        func(OpenAIProviderConfig) (Provider, error)
	Ollama     func(OllamaProviderConfig) (Provider, error)
	OpenCode   func(OpenCodeProviderConfig) (Provider, error)
}

RuntimeProviderFactories provides provider constructors used by the runtime composer.

type ThrottledProvider 

type ThrottledProvider struct {
	// contains filtered or unexported fields
}

ThrottledProvider wraps a Provider with a concurrency limiter. All callers share the same semaphore, preventing Ollama (or any serial LLM backend) from being overwhelmed by concurrent requests.

func NewThrottledProvider 

func NewThrottledProvider(inner Provider, maxConcurrent int) *ThrottledProvider

NewThrottledProvider creates a provider that allows at most maxConcurrent simultaneous LLM calls. Additional calls block until a slot is available.

func NewThrottledProviderWithSemaphore 

func NewThrottledProviderWithSemaphore(inner Provider, sem chan struct{}) *ThrottledProvider

NewThrottledProviderWithSemaphore creates a provider that acquires a slot from the supplied semaphore for each call. Several providers may share one semaphore; tests can pass their own channel to observe capacity.

func (*ThrottledProvider) Capacity 

func (t *ThrottledProvider) Capacity() int

Capacity reports the semaphore size.

func (*ThrottledProvider) Complete 

func (t *ThrottledProvider) Complete(ctx context.Context, request CompletionRequest) (*CompletionResponse, error)

Source Files

View all Source files

Directories

Path Synopsis
Package embedding provides a Provider interface and implementations for generating vector embeddings from text.
 Package embedding provides a Provider interface and implementations for generating vector embeddings from text.
Package opencode implements an LLM provider backed by a private OpenCode server.
 Package opencode implements an LLM provider backed by a private OpenCode server.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.