IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

AutoModel

AutoModel​

class max.pipelines.AutoModel(model_path, *, max_batch_size=1, devices=None, device_graph_capture=False, prefer_module_v3=True, eager=False, **config_overrides)

source

Bases: object

Loads a text or image generation model in-process.

Unlike Hugging Face’s base-model AutoModel, this one generates. The Hugging Face equivalent is AutoModelForCausalLM. Serving features are tested through max serve.

An image generation model, such as FLUX.2, runs through generate() with prompt. Its pipeline takes one prompt per call, so max_batch_size stays 1 and device graph capture does not apply.

model = AutoModel("black-forest-labs/FLUX.2-klein-4B")
result = model.generate(
    prompt=["a red fox in snow", "a lighthouse at dusk"],
    height=512,
    width=512,
    seed=0,
)
result.images[0][0].save("fox.png")

The config turns off the overlap scheduler, device graph capture, prefix caching, the vision encoder cache and chunked prefill. On AMD it also turns off the vendor BLAS fallback for a text model, as serve does with capture on, so matmuls compile to serve’s kernels. As in serve, the architecture’s required_arguments win over the overrides.

References to the pipeline, its KV cache manager, an open GenerateIterator, or a live traceback from inside the pipeline keep device memory alive past close().

Parameters:

  • model_path (str) – A Hugging Face repo ID or a local model directory.
  • max_batch_size (int) – The most prompts per data-parallel replica, fixed at construction. With device_graph_capture=True, warmup captures decode graphs for every batch size up to this value, so a larger value costs more warmup time and device memory.
  • devices (str | None) – The devices to load on, written as the CLI’s --devices takes them, such as "cpu", "gpu:0" or "gpu:0,1". Defaults to the first visible device.
  • device_graph_capture (bool) – Builds the overlap pipeline and runs its warmup_graph_capture(), so decode steps replay captured graphs. Steps stay synchronous unless enable_overlap_scheduler=True is also passed.
  • prefer_module_v3 (bool) – Loads the architecture’s ModuleV3 version, built with max.nn, where it has one.
  • eager (bool) – Runs the decoder op by op instead of compiling it, so pdb and print inside a module’s forward() show real values on every step. Each step is much slower, so use it for debugging. It needs a ModuleV3 text generation architecture, on one device or one device per data-parallel replica.
  • **config_overrides (object) – Flat PipelineArgs field names, such as device_specs or enable_prefix_caching. They win over the debug defaults.

Raises:

  • TypeError – If both devices and device_specs are passed, if an override passes a runtime, kv_cache or model section, or a config_file, instead of flat fields, or a model_override entry sets one of them whole.
  • ValueError – If the config needs a feature that only max serve runs, such as LoRA, a model that generates neither text nor images, a video model or a block diffusion model, if device_graph_capture=True builds a pipeline that cannot capture graphs, or if an image generation model gets device_graph_capture=True or a max_batch_size other than 1, or if eager=True with a config eager can’t run, such as an architecture without a ModuleV3 decoder, a task other than text generation, more devices than eager runs on, device graph capture, the overlap scheduler, LoRA, speculative decoding or expert parallelism.

close()​

close()

source

Releases the pipeline, after shutting down a text model’s KV cache.

Calling it again does nothing. The pipeline is released even if the shutdown raises.

Return type:

None

generate()​

generate(*, prompt: str | Sequence[str], images: Sequence[Sequence[bytes | Image]] | None = None, negative_prompt: str | Sequence[str] | None = None, height: int | None = None, width: int | None = None, num_inference_steps: int | None = None, guidance_scale: float | None = None, true_cfg_scale: float | None = None, strength: float | None = None, num_images_per_prompt: int = 1, seed: int = 0) → GenerateImagesResult

source

generate(input_ids: Iterable[Sequence[int]] | None = None, attention_mask: Iterable[Sequence[int]] | None = None, *, messages: Sequence[Sequence[TextGenerationRequestMessage | Mapping[str, object]]] | None = None, images: Sequence[Sequence[bytes | Image]] | None = None, chat_template_kwargs: Mapping[str, object] | None = None, max_new_tokens: int, min_new_tokens: int | None = None, temperature: float | None = None, top_k: int | None = None, top_p: float | None = None, min_p: float | None = None, frequency_penalty: float | None = None, presence_penalty: float | None = None, repetition_penalty: float | None = None, ignore_eos: bool = False, stop_token_ids: Sequence[int] | None = None, seed: int = 0, output_logits: bool = False, output_logprobs: int = 0) → GenerateResult

Generates text or images, whichever the loaded model makes.

For a text model, it runs generate() on pipeline with the same arguments, so model.generate(**tokenizer(prompts), max_new_tokens=16) works with an AutoTokenizer. Pass messages and images instead of input_ids for chat or image prompts, and read each prompt’s length from the result’s prompt_lengths. chat_template_kwargs applies to every messages row, such as {"enable_thinking": False}.

For an image model, pass prompt and the image arguments, and it runs generate_images() with them. images and seed apply to both, and images holds one list of images per prompt or conversation. An argument counts as passed when it differs from its default.

Raises:

  • TypeError – If a text model gets no max_new_tokens or an image model no prompt, or as described for the function it runs.
  • ValueError – If an argument is for the other kind of model, or as described for the function it runs.
  • RuntimeError – If the model has been closed, or as described for the function it runs.
  • OSError – If an image model’s input image does not decode, as described for generate_images().

generate_iter()​

generate_iter(input_ids=None, attention_mask=None, *, messages=None, images=None, chat_template_kwargs=None, max_new_tokens, min_new_tokens=None, temperature=None, top_k=None, top_p=None, min_p=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, ignore_eos=False, stop_token_ids=None, seed=0, output_logits=False, output_logprobs=0)

source

Opens a GenerateIterator on pipeline.

Takes the same keyword arguments. Use it as a context manager so its rows are released.

Raises:

Parameters:

Return type:

GenerateIterator

pipeline​

property pipeline: TextGenerationPipelineInterface[TextContext]

source

The loaded text pipeline.

Raises:

pipeline_config​

property pipeline_config: PipelineConfig

source

The resolved config that built the pipeline.