For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
AutoModel
AutoModel
class max.pipelines.AutoModel(model_path, *, max_batch_size=1, devices=None, device_graph_capture=False, prefer_module_v3=True, eager=False, **config_overrides)
Bases: object
Loads a text or image generation model in-process.
Unlike Hugging Face’s base-model AutoModel, this one generates. The
Hugging Face equivalent is AutoModelForCausalLM. Serving features are
tested through max serve.
An image generation model, such as FLUX.2, runs through
generate() with prompt. Its pipeline takes one prompt per
call, so max_batch_size stays 1 and device graph capture does not
apply.
model = AutoModel("black-forest-labs/FLUX.2-klein-4B")
result = model.generate(
prompt=["a red fox in snow", "a lighthouse at dusk"],
height=512,
width=512,
seed=0,
)
result.images[0][0].save("fox.png")The config turns off the overlap scheduler, device graph capture, prefix
caching, the vision encoder cache and chunked prefill. On AMD it also
turns off the vendor BLAS fallback for a text model, as serve does with
capture on, so matmuls compile to serve’s kernels. As in serve, the
architecture’s required_arguments win over the overrides.
References to the pipeline, its KV cache manager, an open
GenerateIterator, or a live traceback
from inside the pipeline keep device memory alive past close().
-
Parameters:
-
- model_path (str) – A Hugging Face repo ID or a local model directory.
- max_batch_size (int) – The most prompts per data-parallel replica, fixed at
construction. With
device_graph_capture=True, warmup captures decode graphs for every batch size up to this value, so a larger value costs more warmup time and device memory. - devices (str | None) – The devices to load on, written as the CLI’s
--devicestakes them, such as"cpu","gpu:0"or"gpu:0,1". Defaults to the first visible device. - device_graph_capture (bool) – Builds the overlap pipeline and runs its
warmup_graph_capture(), so decode steps replay captured graphs. Steps stay synchronous unlessenable_overlap_scheduler=Trueis also passed. - prefer_module_v3 (bool) – Loads the architecture’s ModuleV3 version, built
with
max.nn, where it has one. - eager (bool) – Runs the decoder op by op instead of compiling it, so
pdbandprintinside a module’sforward()show real values on every step. Each step is much slower, so use it for debugging. It needs a ModuleV3 text generation architecture, on one device or one device per data-parallel replica. - **config_overrides (object) – Flat
PipelineArgsfield names, such asdevice_specsorenable_prefix_caching. They win over the debug defaults.
-
Raises:
-
- TypeError – If both
devicesanddevice_specsare passed, if an override passes aruntime,kv_cacheormodelsection, or aconfig_file, instead of flat fields, or amodel_overrideentry sets one of them whole. - ValueError – If the config needs a feature that only
max serveruns, such as LoRA, a model that generates neither text nor images, a video model or a block diffusion model, ifdevice_graph_capture=Truebuilds a pipeline that cannot capture graphs, or if an image generation model getsdevice_graph_capture=Trueor amax_batch_sizeother than 1, or ifeager=Truewith a config eager can’t run, such as an architecture without a ModuleV3 decoder, a task other than text generation, more devices than eager runs on, device graph capture, the overlap scheduler, LoRA, speculative decoding or expert parallelism.
- TypeError – If both
close()
close()
Releases the pipeline, after shutting down a text model’s KV cache.
Calling it again does nothing. The pipeline is released even if the shutdown raises.
-
Return type:
-
None
generate()
generate(*, prompt: str | Sequence[str], images: Sequence[Sequence[bytes | Image]] | None = None, negative_prompt: str | Sequence[str] | None = None, height: int | None = None, width: int | None = None, num_inference_steps: int | None = None, guidance_scale: float | None = None, true_cfg_scale: float | None = None, strength: float | None = None, num_images_per_prompt: int = 1, seed: int = 0) → GenerateImagesResult
generate(input_ids: Iterable[Sequence[int]] | None = None, attention_mask: Iterable[Sequence[int]] | None = None, *, messages: Sequence[Sequence[TextGenerationRequestMessage | Mapping[str, object]]] | None = None, images: Sequence[Sequence[bytes | Image]] | None = None, chat_template_kwargs: Mapping[str, object] | None = None, max_new_tokens: int, min_new_tokens: int | None = None, temperature: float | None = None, top_k: int | None = None, top_p: float | None = None, min_p: float | None = None, frequency_penalty: float | None = None, presence_penalty: float | None = None, repetition_penalty: float | None = None, ignore_eos: bool = False, stop_token_ids: Sequence[int] | None = None, seed: int = 0, output_logits: bool = False, output_logprobs: int = 0) → GenerateResult
Generates text or images, whichever the loaded model makes.
For a text model, it runs generate()
on pipeline with the same arguments, so
model.generate(**tokenizer(prompts), max_new_tokens=16) works with
an AutoTokenizer. Pass messages and images instead of
input_ids for chat or image prompts, and read each prompt’s length
from the result’s prompt_lengths. chat_template_kwargs applies
to every messages row, such as {"enable_thinking": False}.
For an image model, pass prompt and the image arguments, and it
runs generate_images() with
them. images and seed apply to both, and images holds one
list of images per prompt or conversation. An argument counts as
passed when it differs from its default.
-
Raises:
-
- TypeError – If a text model gets no
max_new_tokensor an image model noprompt, or as described for the function it runs. - ValueError – If an argument is for the other kind of model, or as described for the function it runs.
- RuntimeError – If the model has been closed, or as described for the function it runs.
- OSError – If an image model’s input image does not decode, as
described for
generate_images().
- TypeError – If a text model gets no
generate_iter()
generate_iter(input_ids=None, attention_mask=None, *, messages=None, images=None, chat_template_kwargs=None, max_new_tokens, min_new_tokens=None, temperature=None, top_k=None, top_p=None, min_p=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, ignore_eos=False, stop_token_ids=None, seed=0, output_logits=False, output_logprobs=0)
Opens a GenerateIterator on pipeline.
Takes the same keyword arguments. Use it as a context manager so its rows are released.
-
Raises:
-
- TypeError – As described for
GenerateIterator. - ValueError – If the model generates images, or as described for
GenerateIterator. - RuntimeError – If the model has been closed, or as described for
GenerateIterator.
- TypeError – As described for
-
Parameters:
-
- input_ids (Iterable[Sequence[int]] | None)
- attention_mask (Iterable[Sequence[int]] | None)
- messages (Sequence[Sequence[TextGenerationRequestMessage | Mapping[str, object]]] | None)
- images (Sequence[Sequence[bytes | Image]] | None)
- chat_template_kwargs (Mapping[str, object] | None)
- max_new_tokens (int)
- min_new_tokens (int | None)
- temperature (float | None)
- top_k (int | None)
- top_p (float | None)
- min_p (float | None)
- frequency_penalty (float | None)
- presence_penalty (float | None)
- repetition_penalty (float | None)
- ignore_eos (bool)
- stop_token_ids (Sequence[int] | None)
- seed (int)
- output_logits (bool)
- output_logprobs (int)
-
Return type:
pipeline
property pipeline: TextGenerationPipelineInterface[TextContext]
The loaded text pipeline.
-
Raises:
-
- ValueError – If the model generates images.
- RuntimeError – If the model has been closed.
pipeline_config
property pipeline_config: PipelineConfig
The resolved config that built the pipeline.