Documentation
¶
Index ¶
- Variables
- type Config
- type FinalLayer
- type FlowNet
- type ResidualBlock
- type State
- type TimeEncoder
- type Transformer
- func (t *Transformer) Advance(x []float32, state *State)
- func (t *Transformer) AdvanceLatent(latent []float32, state *State) []float32
- func (t *Transformer) AdvancePrompt(tokens []int, state *State)
- func (t *Transformer) AdvanceText(token int, state *State) []float32
- func (t *Transformer) AdvanceVoice(latents []float32, frames int, state *State) error
- func (t *Transformer) CanClone() bool
- func (t *Transformer) EOSLogit(cond []float32) float32
- func (t *Transformer) LoadVoice(path string, capacity int) (*State, error)
- func (t *Transformer) NewState(capacity int) *State
- func (t *Transformer) SaveVoice(path string, state *State) error
Constants ¶
This section is empty.
Variables ¶
var ConfigFrench24L = ConfigFor(24)
ConfigFrench24L is the geometry of french_24l, kept as the reference the parity tests are written against.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
transformer.Geometry
LatentDim int // ldim: 32, the dimension of the audio latents
}
Config describes the geometry of the flow_lm, read from the language's YAML.
type FinalLayer ¶
type FinalLayer struct {
Modulation nn.Linear // 512 -> 1024: shift and scale
Linear nn.Linear // 512 -> 32
// contains filtered or unexported fields
}
FinalLayer brings the internal width back down to the latent's dimension.
func (*FinalLayer) Apply ¶
func (c *FinalLayer) Apply(x, cond, out []float32)
type FlowNet ¶
type FlowNet struct {
Time [2]*TimeEncoder
CondEmbed nn.Linear // 1024 -> 512
InputProj nn.Linear // 32 -> 512
Blocks []*ResidualBlock
Final FinalLayer
// contains filtered or unexported fields
}
FlowNet assembles the whole thing.
func LoadFlowNet ¶
LoadFlowNet reads the weights of the flow net.
type ResidualBlock ¶
type ResidualBlock struct {
Modulation nn.Linear // 512 -> 1536: shift, scale, gate
InLN nn.LayerNorm
MLP0, MLP2 nn.Linear
// contains filtered or unexported fields
}
ResidualBlock applies an MLP modulated by the conditioning.
func (*ResidualBlock) Apply ¶
func (b *ResidualBlock) Apply(x, cond []float32)
type State ¶
type State struct {
Caches []*transformer.Cache
}
State is the generation state: one KV cache per layer.
func (*State) Rewind ¶
Rewind brings the state back to an earlier position.
Each segment must restart from the voice as it was, otherwise the second one would inherit the context of the first. Duplicating the state for that would cost four hundred megabytes per segment; rewinding it costs none, because the abandoned positions will be rewritten before being read again — attention never looks beyond the current position.
type TimeEncoder ¶
type TimeEncoder struct {
Freqs []float32 // 128 frequencies, stored in the weights
MLP0 nn.Linear // 256 -> 512
MLP2 nn.Linear // 512 -> 512
Norm nn.RMSNorm
// contains filtered or unexported fields
}
TimeEncoder embeds a scalar instant into a conditioning vector.
func (*TimeEncoder) Apply ¶
func (e *TimeEncoder) Apply(t float32, out []float32)
type Transformer ¶
type Transformer struct {
Config Config
Layers []*transformer.Layer
InputLinear nn.Linear // from the latent of 32 to d_model
OutNorm nn.LayerNorm
OutEOS nn.Linear
EOSBias float32
BOSEmb []float32 // starting latent, substituted for the first frame
EmbMean []float32 // denormalization statistics of the latents
EmbStd []float32
Conditioner [][]float32 // table of text embeddings, indexed by token
// SpeakerProj and BOSBeforeVoice serve the cloning path only: the first
// brings an encoded latent up to d_model, the second is the position the
// model listens from. They are a separate projection from InputLinear —
// listening to a voice and generating from one are not the same operation,
// and the model was not trained as though they were.
SpeakerProj nn.Linear
BOSBeforeVoice []float32
// contains filtered or unexported fields
}
Transformer holds the layers and the input and output projections.
func Load ¶
func Load(m *tensors.Model, cfg Config) (*Transformer, error)
Load reads from the weights file everything the transformer uses.
func (*Transformer) Advance ¶
func (t *Transformer) Advance(x []float32, state *State)
Advance passes one position through every layer — 24 or 6, depending on the language. x is modified in place.
func (*Transformer) AdvanceLatent ¶
func (t *Transformer) AdvanceLatent(latent []float32, state *State) []float32
AdvanceLatent consumes one audio latent: it is projected to d_model, crosses the layers, then goes through the output normalization. The returned vector is the conditioning of the flow net for the next frame.
func (*Transformer) AdvancePrompt ¶
func (t *Transformer) AdvancePrompt(tokens []int, state *State)
AdvancePrompt consumes all the tokens of a segment in one go.
Nothing requires feeding them one by one: they are all known in advance, and processing them together divides the re-reading of the 604 MB of weights by as much. That is the difference between an instant start and half a second of waiting before the first sound.
func (*Transformer) AdvanceText ¶
func (t *Transformer) AdvanceText(token int, state *State) []float32
AdvanceText consumes one text embedding: same path, but the input comes from the conditioner's table rather than from a projected latent.
func (*Transformer) AdvanceVoice ¶ added in v0.3.0
func (t *Transformer) AdvanceVoice(latents []float32, frames int, state *State) error
AdvanceVoice listens to an encoded recording and leaves the state holding the voice, ready to be saved or generated from.
latents are what the Mimi encoder produced, laid out channel by channel: latents[c*frames+f]. They go in together rather than one at a time, for the reason AdvancePrompt gives — the weights are read once instead of once per frame, and a recording is hundreds of frames long.
func (*Transformer) CanClone ¶ added in v0.3.0
func (t *Transformer) CanClone() bool
CanClone says whether the weights carry what listening to a voice needs.
func (*Transformer) EOSLogit ¶
func (t *Transformer) EOSLogit(cond []float32) float32
EOSLogit says how finished the model judges the utterance to be.
func (*Transformer) LoadVoice ¶
func (t *Transformer) LoadVoice(path string, capacity int) (*State, error)
LoadVoice returns a state initialized from the file's caches, sized for `capacity` positions in total.
func (*Transformer) NewState ¶
func (t *Transformer) NewState(capacity int) *State
NewState allocates an empty state for at most `capacity` positions.
func (*Transformer) SaveVoice ¶ added in v0.3.0
func (t *Transformer) SaveVoice(path string, state *State) error
SaveVoice writes the state as a voice file, in the layout LoadVoice reads and the Python daemon wrote: one K/V cache and one offset per layer, in safetensors.
The caches are stored at exactly the length they hold. The reference writes them at their allocated capacity and leaves the rest NaN, which LoadVoice guards against; there is no reason to reproduce a hazard that costs disk space.
Source Files