llmx

type CachingClient ¶

type CachingClient struct {
	// contains filtered or unexported fields
}

CachingClient decorates an llm.Client with a persistent on-disk cache for Embeddings calls and, optionally (WithChatCache), for deterministic ChatCompletion calls. Embedding a text is deterministic for a given model, so the vector can be reused across runs — turning repeated corpus ingestions (evaluation re-runs, benchmarks) into free cache hits instead of thousands of billable, rate-limited calls. A chat completion is only cacheable when the caller pins a seed (e.g. the HyDE transformer, seeded per query for reproducibility): unseeded, tool-using or multimodal calls always pass through.

Each embeddings input is cached individually under sha256(namespace, dimensions, text) below dir/embeddings, so hits survive re-batching. On a batch call only the misses are forwarded to the wrapped client (in one batch), and the reported usage covers those misses only. Chat completions are cached below dir/chat under a key covering the messages, seed, temperature and response schema. Files are written atomically (temp file + rename) making the cache safe for concurrent use; an unreadable or corrupted entry is treated as a miss and rewritten. Concurrent misses on the same key are collapsed into a single call to the wrapped client, so a burst of identical requests costs one round-trip rather than one per caller.

The namespace must identify the embedding space — typically the model name. Reusing a directory with a different model but the same namespace would serve vectors from the wrong space. The chat namespace (WithChatCache) must likewise identify the chat model.

func NewCachingClient(client llm.Client, dir, namespace string, funcs ...CachingOption) (*CachingClient, error)

func (c *CachingClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)

func (c *CachingClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)

func (c *CachingClient) ChatStats() (hits, misses int64)

func (c *CachingClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)

func (c *CachingClient) Stats() (hits, misses int64)

func (c *CachingClient) Transcription(ctx context.Context, audio []byte, funcs ...llm.TranscriptionOptionFunc) (llm.TranscriptionResponse, error)

type CachingOption ¶ added in v0.2.0

type CachingOption func(*CachingClient)

CachingOption configures a CachingClient.

func WithChatCache(namespace string) CachingOption

type ObservableClient ¶

type ObservableClient struct {
	// contains filtered or unexported fields
}

ObservableClient decorates an llm.Client with OpenTelemetry spans and metrics (call count, latency, token usage). It is a thin wrapper: when no OTel provider is installed the instrumentation is a no-op. Compose it with RetryClient as needed (e.g. observe the retried client to count every attempt's latency, or wrap the outside to measure the logical call).

func NewObservableClient(client llm.Client) *ObservableClient

func (c *ObservableClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)

func (c *ObservableClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)

func (c *ObservableClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)

func (c *ObservableClient) Transcription(ctx context.Context, audio []byte, funcs ...llm.TranscriptionOptionFunc) (llm.TranscriptionResponse, error)

type OptionFunc ¶

type OptionFunc func(*Options)

func WithBackoff(base, max time.Duration) OptionFunc

func WithLimiter(limiter *rate.Limiter) OptionFunc

func WithMaxRetries(n int) OptionFunc

func WithRateLimit(r rate.Limit, burst int) OptionFunc

func WithRetryAfter(fn func(error) (time.Duration, bool)) OptionFunc

func WithRetryable(fn func(error) bool) OptionFunc

type Options ¶

type Options struct {
	// MaxRetries is the number of retries attempted after the first failure
	// (so a total of MaxRetries+1 attempts). A negative value disables retries.
	MaxRetries int
	// BaseBackoff is the delay before the first retry; it doubles on each
	// subsequent retry, capped at MaxBackoff. The delay actually waited is
	// drawn from [backoff/2, backoff) — see jitter.
	BaseBackoff time.Duration
	// MaxBackoff caps the backoff delay.
	MaxBackoff time.Duration
	// Retryable decides whether an error is worth retrying. Defaults to
	// DefaultRetryable (everything except context cancellation/deadline).
	Retryable func(error) bool
	// RetryAfter extracts the delay a provider explicitly asked us to wait
	// (an HTTP Retry-After header on a 429/503). When it reports a delay, that
	// delay wins over the computed backoff — the provider knows when its quota
	// window reopens, we only guess. Defaults to DefaultRetryAfter.
	RetryAfter func(error) (time.Duration, bool)
	// Limiter, when set, throttles every call (Wait before each attempt,
	// including retries).
	Limiter *rate.Limiter
}

Options configures a RetryClient.

type RetryAfterError ¶ added in v0.15.0

type RetryAfterError interface {
	error
	// RetryAfter returns the delay to wait before retrying.
	RetryAfter() time.Duration
}

RetryAfterError is implemented by errors carrying a provider-mandated retry delay, typically parsed from an HTTP Retry-After header. Providers whose errors implement it are honoured automatically by DefaultRetryAfter.

type RetryClient ¶

type RetryClient struct {
	// contains filtered or unexported fields
}

RetryClient decorates an llm.Client with retries (exponential backoff) and an optional rate limiter. It is safe for concurrent use as long as the wrapped client is.

func NewRetryClient(client llm.Client, funcs ...OptionFunc) *RetryClient

func (c *RetryClient) ChatCompletion(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (llm.ChatCompletionResponse, error)

func (c *RetryClient) ChatCompletionStream(ctx context.Context, funcs ...llm.ChatCompletionOptionFunc) (<-chan llm.StreamChunk, error)

func (c *RetryClient) Embeddings(ctx context.Context, inputs []string, funcs ...llm.EmbeddingsOptionFunc) (llm.EmbeddingsResponse, error)

func (c *RetryClient) Transcription(ctx context.Context, audio []byte, funcs ...llm.TranscriptionOptionFunc) (llm.TranscriptionResponse, error)