Introduction

Much of the focus in AI engineering over the past few years went into turning models from question-answerers into agents that can do more than talk: planning steps, using tools, remembering context, or working through long, complex tasks in the background. However, the agent stack is still mainly focused on text. Even as foundation models become increasingly multimodal, the field still treats a multimodal agent as a text agent with extra modalities.

As models improve at processing and generating rich multimodal content, they are naturally being pulled into visual and creative work. With that, the demand shifts toward end-to-end deliverables. Users do not want to manually prompt every individual asset, but because creative work is subjective, they rarely want hands-off, black-box automation either. They want systems that take a high-level goal, handle multi-step execution, and keep humans in the loop for review and creative steering.

This is the promise of Generative Media Agents (GenMedia agents): systems designed to orchestrate and iterate on generative media from an objective to a finished deliverable, working alongside human creators.

Building them, however, requires rethinking key assumptions about agent design. Working with media fundamentally changes how an agent tracks state, executes edits, and judges whether an output is actually good enough. This post covers what a GenMedia agent is made of, when to use one, how it differs from a text agent, and how to build one on Google Cloud, along with reference implementations.

Anatomy of a GenMedia agent

Not every media task needs an agent. Generating image variants from a fixed template or running batch voiceovers across a catalog is just a pipeline: predictable, batchable, and simple to test.

You need an agent when the creative path cannot be scripted in advance. These are open-ended workflows where an objective spans multiple models, user feedback directs each step, and generation must be combined with downstream tools and business data.

At that level of complexity, a typical GenMedia agent architecture breaks down into four core layers as shown in Figure 1.

• The interface: where users review outputs and steer the process, across modalities like voice, text, camera or screen.
• The harness: the runtime environment (such as Agent Development Kit (ADK)) that runs the main agentic loop: it manages sessions, tracks state, runs the tools (functions you write, Model Context Protocol (MCP) servers, skills, other agents) and delivers the results.
• The model: the cognitive layer (like Gemini) that interprets creative direction, decides when to call a tool and writes its prompt.
• External systems: what the tools call, including the generative media models, search grounding, object storage and enterprise business services.

Levels of abstraction

Knowing the components is one thing; deciding how much of that harness to write yourself is the practical choice.

Teams implement GenMedia agents across three levels of abstraction, depending on how much control they need over the runtime.

Level 1: Model-orchestrated tools (code-first): You build the runtime loop and interface yourself, using the model for reasoning and function calling to trigger media tools.

• When to use: Low-latency interactive experiences (like voice-driven audio/visual co-creation) and lightweight single-purpose apps where you need direct control over UI delivery.
• Typical users: Creative technologists and frontend/full-stack engineers.

Level 2: Agent frameworks (managed harness): You build on a standardized harness, for instance using ADK. The framework handles session lifecycles, long-running video tasks, MCP integrations and multi-agent patterns. In ADK terms, that is an LlmAgentwith a generation tool wrapped as a LongRunningFunctionTool, the GenMedia MCP servers attached through an McpToolset, and a SkillToolsetthat loads prompting guides on demand.

root_agent = LlmAgent(
    model="gemini-...",
    tools=[
        LongRunningFunctionTool(generate_video),
        McpToolset(connection_params=genmedia_mcp),
        SkillToolset(skills=[load_skill_from_dir("skills/omni-prompting")]),
    ],
)

• When to use: Production creative pipelines requiring persistent state, asynchronous background generation, multi-agent orchestration, and enterprise scale.
• Typical users: Backend AI engineers and production platform teams.

Level 3: Visual and no-code builders: You assemble workflows visually on a canvas like Agent Studio, Google Flow, or ComfyUI, relying on the platform to manage execution and routing.

• When to use: Creative exploration, rapid concept validation, and node-based media experimentation.
• Typical users: Creative directors, technical artists, and product teams prototyping flows without writing code.

To see how the three abstraction levels in Figure 2 map to actual implementation effort, the matrix below highlights what you build yourself versus what the framework or platform provides. At level 1 you own everything except the models. At level 2 the framework owns sessions, tool execution and MCP, and you own the media plane, the asset registry and the interface. At level 3 the platform owns the harness and interface, and you own prompts and tool configuration.

Six operational constraints of generative media agents

At a high level, the architectural layers resemble those of a conventional text agent. What changes is what flows through them: media generation introduces six operational constraints that shape harness design, model reasoning, and external integrations:

1. Latency: Generating an image takes seconds, while rendering video takes tens of seconds to minutes. The harness must support asynchronous, long-running tool execution (such as ADK’s LongRunningFunctionTool) so conversational streaming over the Gemini Live API continues uninterrupted while generation jobs run in the background.
2. Output size: Results are heavy binary payloads (megabytes of raw image arrays or MP4 streams). The harness routes binary data directly to object stores like Google Cloud Storage and straight to the client interface, passing only lightweight URIs and metadata back to the model context.
3. State and lineage: Creative workflows are iterative. Users continually reference earlier iterations (“make the second shot darker,” “extend that clip”), requiring the harness session service or external asset registries to track asset lineage, prompt history, and parent-child dependencies across turns.
4. Prompt engineering: Raw user requests like “a car driving at night” require expansion into technical visual language covering camera movement, focal length, lighting, and composition. Generative models expect cinematic shot grammar, so this prompting expertise must live in the agent’s instructions or dynamically retrieved skill documents.
5. Cost and execution limits: Media calls carry higher per-invocation costs than simple token completions. The harness requires strict concurrency caps, retry limits, and confirmation checkpoints before triggering expensive multi-model tasks.
6. Safety, provenance, and regional compliance: Media endpoints enforce prompt safety filters and regional availability constraints, and apply provenance signals such as SynthID watermarking from Google DeepMind and, where supported, C2PA Content Credentials.

Design patterns for GenMedia agents

Across all three implementation levels, the following design patterns help solve these six constraints in production:

1. Separate your media plane from the control plane

Passing raw binary buffers (megabytes of image arrays or video streams) through an LLM’s context window exhausts token limits and increases latency.

Treat your agent strictly as the control plane. It only needs to see lightweight references: asset IDs, media types, and cloud storage URIs. Leave the heavy lifting to your media plane (object storage and edge delivery), which routes the actual bytes between generation endpoints and your UI. Here is what that looks like in a typed tool return:

from typing import Literal, Optional
from pydantic import BaseModel

class MediaReference(BaseModel):
    """Registered in the media plane once the worker finishes rendering."""
    asset_id: str
    storage_uri: str  # Direct Cloud Storage URI; raw bytes bypass the agent
    media_type: Literal["image/png", "video/mp4"]

def get_asset(asset_id: str) -> MediaReference:
    """The agent sees a URI and metadata, never the bytes."""
    return asset_registry.lookup(asset_id)

2. Decouple generation jobs from the chat loop

Video and high-resolution image generation can take anywhere from 30 seconds to a couple of minutes. If you wait synchronously, your whole conversational loop freezes, network timeouts kick in, and the user is left staring at a dead screen.

Instead, decouple the generation job from the conversational turn:

• Return an immediate acknowledgement (like "status": "in_progress") so your agent can confirm receipt and keep talking.
• Push the actual rendering task to a background worker queue.
• Once the render completes, push the asset straight to the UI out-of-band and drop a lightweight completion notice back into the session history.

def generate_video(
    prompt: str,
    duration_seconds: int = 8,
    aspect_ratio: Literal["16:9", "9:16"] = "16:9",
    reference_asset_id: Optional[str] = None,
) -> dict:
    """Queues a render and returns a receipt so the conversation keeps going."""
    job_id = task_queue.enqueue(
        "render_worker",
        prompt=prompt,
        duration=duration_seconds,
        ratio=aspect_ratio,
        parent_id=reference_asset_id,
    )
    return {"job_id": job_id, "status": "in_progress", "estimated_seconds": 90}


def render_worker(job_id: str, session_id: str, **params) -> None:
    """Runs off the chat loop. Renders, registers, delivers, then notifies."""
    video_bytes = video_model.generate(**params)
    storage_uri = media_store.put(video_bytes, content_type="video/mp4")
    asset = asset_registry.register(
        storage_uri=storage_uri,
        media_type="video/mp4",
        parent_id=params.get("parent_id"),
        prompt=params["prompt"],
    )
    ui.push(session_id, asset)
    session_service.append_event(
        session_id,
        {"job_id": job_id, "status": "done", "asset_id": asset.asset_id},
    )

3. Track state and lineage in your harness

Creative work is rarely linear; it is an iterative, branching tree. When someone says, “Go back to the second version, but make the background sunset,” your agent needs to know exactly which asset they mean and what parameters created it.

Do not rely on raw chat memory for this. Keep a stateful asset registry inside your harness that maps child assets back to their parent IDs, original prompts, seeds, and generation parameters. This is what lets you build non-destructive rollbacks, branch comparisons, and reliable audit logs.

4. Compile and refine prompts in the agent

Most users will not talk like film directors or seasoned prompt engineers. They will say things like “make a cool car clip.” But generative media models need precise, technical language to produce stable results.

Your orchestrating agent should act as a prompt compiler. Teach it visual and cinematic grammar in its instructions so it can translate casual requests into structured parameters before hitting your APIs. For example:

• For video: Specify camera movement (dolly, crane, tracking), focal depth, lighting style, and explicit anchors like “single continuous shot” so the video model does not splice in unwanted cuts.
• For images: Specify composition, perspective, rendering style, and exact quoted text for in-image typography.

5. Evaluate across three distinct tiers

When a media agent breaks, it rarely fails in a single, predictable way. You need three separate testing layers to catch problems before users do:

• Dispatch accuracy (The routing layer): Does the model select the right tool with the right arguments? The check is deterministic and cheap to run, and it immediately catches regressions when someone tweaks a docstring.
• Semantic asset quality (The visual layer): Did the model produce a good image or video? Use multimodal LLM judges and human spot-checks against clear rubrics (prompt adherence, physical artifacts, text readability, temporal stability).
• Pipeline latency (The transport layer): Where is time actually being spent? Track your time-to-first-acknowledgement separately from your final asset delivery so you know whether bottlenecks come from the LLM, the rendering queue, or asset transfer.

6. Put guardrails around cost and refusals

Cap concurrent generation jobs per session, set retry limits, and require an explicit confirmation step before any multi-model chain. Catch upstream safety refusals and filter blocks in the tool wrapper and return them to the model as a structured result, so the agent can explain the issue and suggest a rephrased prompt without crashing the session.

Two reference implementations

This section provides two open-source projects that demonstrate these patterns end to end. Both run as real-time agents on the Gemini Live API.

1. GenMedia Live

A level 1 agent built on the Gemini Live API that integrates Nano Banana for image generation, Gemini Omni for video generation, and ffmpeg as tools. Users can generate and edit images by voice, animate stills into video, assemble clips into short films, and use active camera or screen shares as visual references.

2. Live Avatar Studio

A level 2 agent for interactive avatars. It streams video of a speaking avatar instead of plain audio, pairing image editing and video generation tools for dynamic scene composition. Users can share screens or camera feeds as visual references while customizing avatar styles and voices.

Conclusion

Building reliable GenMedia agents today still means writing the pieces no framework gives you: the asset registry, the media plane, the async job queue, and the branching edit history.

While the developer tooling is still catching up, the applications are already here. Teams are already deploying voice-guided visual editors, live-streaming avatars, and automated multimodal production workflows. Solid systems engineering is what keeps those experiences fast, resilient, and cost-effective at scale.

👉 For a hands-on version of the evaluation and prompting patterns above, see AI Agents for Image and Video Generation, the DeepLearning.AI short course I co-teach.

📜About the author

Dr. Wafae Bakkali is a Tech Lead AI, Blackbelt at Google. With over 13 years of experience, she has a proven track record of combining deep technical expertise and leadership with strategic business insights to drive impactful outcomes. Through her current role at Google and former roles at reputable organizations such as AWS, Dr. Bakkali has successfully guided organizations across various industries in adopting and building AI solutions.

She holds a PhD from IMT Atlantique, France, and served as a postdoctoral researcher at Centrale Supélec, Paris-Saclay University, France.

Dr. Bakkali is also an active contributor to the AI community. She regularly shares her expertise as a speaker at international tech conferences and mentors participants in AI hackathons, tech, and startup events.

🔗Connect with me on LinkedIn: https://www.linkedin.com/in/wafae-bakkali/

Building generative media agents: model, harness, tools was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Voice-driven multimodal creation built on Gemini Live API with image and video generation.

👉 The complete code for this workflow is available on GitHub.

Introduction

Generative AI multimodal capabilities are rapidly improving, enabling high-quality generation across images, video, voice, and beyond. By combining multiple modalities, multimodal AI systems are unlocking entirely new kinds of experiences across domains, from voice assistants and conversational interfaces to applications for creative media.

One of the key reasons behind the rapid adoption of these systems is accessibility. AI models are becoming easier to use because we can communicate with them in natural language and interact through multiple, more intuitive ways.

Today, we still do not have a single model that supports all multimodal input–output combinations, such as audio-to-image generation. A practical way to unlock richer multimodal experiences is to build workflows that combine multiple models, each optimized for a specific capability.

One particularly interesting direction is the combination of real-time voice with image and video creation, where voice becomes the control layer for creation. Through natural voice interaction, users can generate images, produce videos, and even assemble creative workflows such as short films.

This approach is especially compelling for use cases like storytelling, creative media production, and content creation, where ideas evolve through iteration rather than fixed inputs.

This post explores that direction through GenMedia Live, a workflow that combines Google’s Gemini Live API, Gemini 3 Pro Image (aka Nano Banana Pro), and Veo to enable multimodal creation through voice.

High-Level Overview

GenMedia Live uses Gemini Live API for live conversational interaction and orchestration, extending its capabilities through function calling to:

• Gemini 3 Pro Image for image generation and editing
• Veo for video generation
• ffmpeg for frame extraction and video combining

You can interact naturally with voice or text to generate images from descriptions or edit existing ones, create videos from text or animate generated images, combine videos into short movies, and show your camera or share your screen to ask questions about what you see.

How It Works

Let’s look at how GenMedia Live works in practice.

👉 Note: Code samples are simplified to illustrate the pattern. See the code repo for the full implementation.

Real-Time Voice Streaming

Real-time voice streaming is based on Gemini Live API. For details on setting up real-time streaming with Gemini Live, see Google’s Live API documentation and cookbook examples.

Defining Tools for Gemini Live API

The key pattern is registering tools that Gemini Live API can call. Each tool has a name, description, and parameter schema. The descriptions matter because Gemini Live API reads them to understand what each tool does and decides which one to call based on the user’s request. Here’s an example of how the tools are defined:

from google.genai import types
tools = [
    types.Tool(function_declarations=[
        types.FunctionDeclaration(
            name="generate_image",
            description="Generate or edit an image",
            parameters={
                "type": "object",
                "properties": {
                    "prompt": {"type": "string"},
                    "refers_to_last": {"type": "boolean"},
                },
                "required": ["prompt"]
            }
        ),
        types.FunctionDeclaration(
            name="generate_video",
            description="Generate a video from text or animate an image",
            parameters={
                "type": "object",
                "properties": {
                    "prompt": {"type": "string"},
                    "duration": {"type": "integer"},
                    "refers_to_last": {"type": "boolean"},
                },
                "required": ["prompt"]
            }
        ),
        types.FunctionDeclaration(
            name="extract_frame",
            description="Extract a frame from a video at a specific timestamp",
            parameters={...}
        ),
        types.FunctionDeclaration(
            name="combine_videos",
            description="Combine multiple videos into one",
            parameters={...}
        ),
    ])
]

Connecting to Gemini Live API

Once the tools are defined, we connect to Gemini Live API and pass the tools in the configuration. We also enable session resumption to maintain conversation history, allowing users to reference previous generations (e.g. “edit image 2”, “animate the last one”, etc). Here’s how to set up the connection:

from google import genai
client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
config = types.LiveConnectConfig(
    response_modalities=["AUDIO"],
    system_instruction="You are GenMedia Live, a creative AI assistant...",
    session_resumption=types.SessionResumptionConfig(handle=resumption_handle),
    tools=tools
)
async with client.aio.live.connect(
    model="gemini-live-2.5-flash-native-audio", 
    config=config
) as session:
    # Handle real-time interaction

Handling Function Calls

When a user makes a request like “generate an image of a dragon” or “combine all videos,” Gemini Live API decides which tool to call and sends a function call with the tool name and arguments. Our code then routes to the appropriate handler based on the function name:

async for response in session.receive():
    if response.tool_call:
        for fc in response.tool_call.function_calls:
            if fc.name == "generate_image":
                result = handle_image_generation(dict(fc.args))
            elif fc.name == "generate_video":
                result = handle_video_generation(dict(fc.args))
            elif fc.name == "extract_frame":
                result = handle_extract_frame(dict(fc.args))
            elif fc.name == "combine_videos":
                result = handle_combine_videos(dict(fc.args))
            
            await session.send_tool_response(function_responses=[
                types.FunctionResponse(id=fc.id, name=fc.name, response=result)
            ])

Image Generation with Gemini 3 Pro Image

The generate_image handler calls Gemini Pro Image to create or edit images. Users can reference a previously generated image (e.g. "edit the last one") or use images from their camera, screen share, or upload images from the UI as reference:

def handle_image_generation(args):
    prompt = args.get("prompt")
    refers_to_last = args.get("refers_to_last", False)
    
    content_parts = []
    
    # Add reference image if editing
    if refers_to_last and last_generated_image:
        content_parts.append({
            "inline_data": {"mime_type": "image/png", "data": last_generated_image}
        })
        content_parts.append(f"Based on this image: {prompt}")
    else:
        content_parts.append(prompt)
    
    # Call Gemini 3 Pro Image
    response = image_client.models.generate_content(
        model="gemini-3-pro-image-preview",
        contents=content_parts,
        config=types.GenerateContentConfig(response_modalities=['TEXT', 'IMAGE'])
    )
    
    # Save and return result

Video Generation with Veo 3.1

The generate_video handler calls Veo to create videos. When a user speaks, Gemini Live API processes the audio, understands the request, and passes the prompt as text in the function call arguments. Users can also animate a previously generated image by referencing it (e.g. "animate the last image"):

def handle_video_generation(args):
    prompt = args.get("prompt")
    duration = args.get("duration", 8)
    refers_to_last = args.get("refers_to_last", False)
    
    config = types.GenerateVideosConfig(
        aspect_ratio="16:9",
        duration_seconds=duration,
        enhance_prompt=True,
    )
    
    # With or without reference image
    if refers_to_last and last_generated_image:
        operation = video_client.models.generate_videos(
            model="veo-3.1-generate-001",
            prompt=prompt,
            image=types.Image(image_bytes=last_generated_image, mime_type="image/png"),
            config=config
        )
    else:
        operation = video_client.models.generate_videos(
            model="veo-3.1-generate-001",
            prompt=prompt,
            config=config
        )
    
    # Poll for completion and save result

Frame Extraction and Video Combining

GenMedia Live also includes utility tools powered by ffmpeg. The extract_frame handler extracts a frame from a video at a specific timestamp, and the combine_videos handler merges multiple videos into one, enabling users to create short movies from their generated clips:

def handle_extract_frame(args):
    video_id = args.get("video_id")
    timestamp = args.get("timestamp", 0)
    
    video_path = f"outputs/videos/video_{video_id}.mp4"
    frame_path = f"outputs/images/image_{next_id}.png"
    
    # Extract frame using ffmpeg
    subprocess.run([
        "ffmpeg", "-ss", str(timestamp),
        "-i", video_path,
        "-vframes", "1",
        frame_path
    ])
    
    # Save and return result

def handle_combine_videos(args):
    video_ids = args.get("video_ids", [])
    output_path = f"outputs/videos/video_{next_id}.mp4"
    
    # Create file list for ffmpeg concat
    with open("concat_list.txt", "w") as f:
        for vid_id in video_ids:
            f.write(f"file 'outputs/videos/video_{vid_id}.mp4'\n")
    
    # Combine videos using ffmpeg
    subprocess.run([
        "ffmpeg", "-f", "concat",
        "-safe", "0",
        "-i", "concat_list.txt",
        "-c", "copy",
        output_path
    ])
    
    # Save and return result

The Complete Flow

For each interaction:

1. User speaks → audio streams to Gemini Live API
2. Gemini Live API understands intent → decides to call a tool
3. Tool call received → routed to handler
4. Handler calls Gemini Pro Image, Veo, or ffmpeg → saves output
5. Result sent back to Gemini Live API → speaks confirmation to user

The session maintains conversation history, allowing Gemini Live API to recall the context of previous generations, while the application layer manages the media assets for iterative editing.

Demo: GenMedia Live in Action

See how natural voice interaction drives image and video generation, enabling multimodal creation as a single conversational flow.

👉 Note: For demonstration purposes, the wait times between the voice command and the final image and video output have been accelerated.

Try It Yourself

To run GenMedia Live, you’ll need:

• Python 3.10+
• A Google Cloud project with Vertex AI API enabled
• ffmpeg installed (installation guide)

Clone and run:

git clone https://github.com/WafaeBakkali/genmedia-live.git
cd genmedia-live
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python app.py

Open http://localhost:8080, authenticate with your Google Cloud credentials, and click Voice to start.

Example use cases to try with GenMedia Live include:

• Storytelling and creative media, combining generated images and videos into short, conversationally driven narratives
• Virtual try-on using a live camera feed to explore outfits, styles, and short animated previews through voice
• Home decoration and interior design based on what the camera sees, including furniture changes and style transformations
• Educational and explainer content, where spoken questions are turned into images, animations, or short videos

Conclusion

GenMedia Live demonstrates how to extend Gemini Live API’s real-time conversation capabilities to image and video generation through function calling. By combining Gemini Live API, Gemini 3 Pro Image, Veo, and ffmpeg, users can create and edit images, generate videos, and produce short movies through natural voice and text interaction.

This type of interaction can enable new possibilities across many domains such as creative workflows where ideas evolve through conversation, interactive learning experiences, or accessible interfaces for people who prefer voice over typing. Some ideas to extend this pattern include:

• Add more tools (music generation, code generation, etc.)
• Build domain-specific applications (storyboarding, product visualization, education)
• Integrate with external APIs (search, databases, file storage)

Disclaimer

This project is a technical proof-of-concept for educational purposes. This is not an officially supported Google product.

📜About the author

Dr. Wafae Bakkali is a Staff Generative AI Specialist, Blackbelt at Google. With over 13 years of experience, she has a proven track record of combining deep technical expertise and leadership with strategic business insights to drive impactful outcomes. Through her current role at Google and former roles at reputable organizations such as AWS, Dr. Bakkali has successfully guided organizations across various industries in adopting and building AI solutions.

She holds a PhD from IMT Atlantique, France, and served as a postdoctoral researcher at Centrale Supélec, Paris-Saclay University, France.

Dr. Bakkali is also an active contributor to the AI community. She regularly shares her expertise as a speaker at international tech conferences and mentors participants in AI hackathons, tech, and startup events.

🔗Connect with me on LinkedIn: https://www.linkedin.com/in/wafae-bakkali/

GenMedia Live: Voice-Driven Multimodal AI Creation with Gemini Live API, Gemini 3 Pro Image, and… was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Exploring a generative AI workflow that combines computer use, multimodal reasoning, and creative generation to build end-to-end shopping experiences.

👉 The complete code for this workflow is available on GitHub.

Introduction

Generative AI has transformed how we create, learn, and connect, but also how we shop. Online shopping often means endless scrolling, filtering, and comparing. Despite having infinite options, finding what we actually want can still take hours.

But what if shopping were conversational? What if you could simply say: “I need an outfit for a birthday party,” or “I want to decorate my living room in a cozy, minimalist style.” and get back curated selections, a photorealistic preview showing you wearing the outfit or your room styled with new furniture, and even a short video that brings it all to life?

In this post, we’ll explore what becomes possible when Gemini 2.5 Computer Use, Gemini 2.5 Flash Image (aka Nano Banana), and Veo 3.1 work together to power an AI shopping workflow that can:

• Understand a user’s natural language intent and break it down into a list of structured, actionable search terms
• Automatically perform multiple searches on the web to collect relevant products for each term
• Generate realistic creative visuals and videos showing how items fit together

The AI Shopping Workflow: An Overview

While our main example will be finding an outfit, this exact workflow can be applied to other use cases, like home decor, event planning, and more.

Here’s the global picture of what we’re building:

Together, these three models form a multimodal pipeline:

• Gemini 2.5 Computer Use → Web navigation and data collection
• Gemini 2.5 Flash Image (Nano Banana) → Image synthesis and virtual try-on
• Veo 3.1 → Video generation and realistic motion

Step 1: The Agentic Loop: Navigate Websites (Gemini 2.5 Computer Use)

Once the user’s intent is broken down, the next task is interacting with the web to find the right data. This is where Gemini 2.5 Computer Use comes in.

What is Gemini 2.5 Computer Use?

In the context of Artificial Intelligence (AI), “computer use” refers to an agent’s ability to operate a computer’s graphical user interface (GUI) to achieve a goal. Instead of relying on structured data or APIs, this type of agent “sees” the screen visually and decides where to click, type, and scroll to complete a task. Think of it as an AI agent that can operate a web browser by seeing the screen and using a virtual “mouse” and “keyboard” to execute a task.

Gemini 2.5 Computer Use is Google’s specialized model built to do exactly this. It’s built on Gemini 2.5 Pro’s capabilities to power agents capable of interacting with user interfaces.

It works as part of an agentic loop process:

1. Input: The model receives a user request (the goal) and a screenshot of the current environment.
2. Analysis: It analyzes these inputs and generates a response, typically a function_call representing a UI action (e.g., click_at, type_text_at).
3. Action: Your application code executes this action.
4. Feedback: A new screenshot is captured and sent back to the model, restarting the loop.
5. This iterative process continues until the task is complete.

In our AI shopping workflow, this agentic loop is the engine for the search step. It’s what turns the user’s intent into action on the web. When a user gives a complex prompt, like: “I need an outfit for a job interview under 300 euros”.

The workflow first uses Gemini 2.5 Flash model to decompose this query into a list of searchable items (e.g., blazer, shirt, pants, shoes)

Once we have this list, the Computer Use agent takes over. For each item in the list, it performs an independent, automated browsing session. It will:

• Navigate to an e-commerce site (using permitted, test-friendly environments).
• Click the cookie acceptance button.
• Type “blazer” into the search bar and press Enter.
• Scroll down the page to load more results.
• Extract the product details (name, price, image URL) from the 5 best-matching items.
• Repeat this entire process for “shirt,” “ pants,” and “shoes.”

The image below shows our application’s logs, which capture the series of actions and observations as the model works towards its goal.

The code to run this is straightforward. We operate in a loop: we give the agent our goal and a screenshot, it gives us the next action to perform, we execute that action, and repeat.

# From computer_use_helper.py - Main Action Loop
# The user's goal
goal = f"""Search Amazon.fr for "{search_terms}".
1. Click search box
2. Type: "{search_terms}"
3. Press Enter
4. Scroll down twice
5. Say "Done"
"""
# Give the goal and the first screenshot
contents = [
    Content(
        role="user",
        parts=[
            Part(text=goal),
            Part.from_bytes(data=screenshot, mime_type="image/png")
        ]
    )
]
# Loop until the agent is "Done"
for turn in range(max_turns):
    # Get the agent's desired action
    response = client.models.generate_content(
        model=MODEL, # "gemini-2.5-computer-use-preview-10-2025"
        contents=contents,
        config=config # Config specifies the computer_use tool
    )
    
    candidate = response.candidates[0]
    
    # Omitted for brevity
    
    # Execute the browser actions (click, type, scroll)
    results, fcs, parts = execute_actions(candidate, page, ...)
    
    # Create the function response and new screenshot
    function_responses, screenshot = create_function_responses(page, results, fcs, parts)
    
    # Omitted for brevity

At the end of this step, our workflow has a curated list of 20 products (5 for each of the 4 items) ready for the user to review.

👉 Note: For this demonstration, we are targeting a single, known e-commerce site for simplicity and reliability. However, Gemini 2.5 Computer use itself is site-agnostic, and this workflow could be generalized to navigate other sites.

For the workflow’s design, the code provides a specific, multi-step goal (e.g., "Click search box," "Type," "Scroll") and programmatically navigates to a known e-commerce site.

This defines the high-level strategy for the agent. The computer use model’s job is to take each of those steps and figure out how to visually execute it, for example, by looking at the screenshot to find the actual coordinates of the “search box” to click or identifying where to type the text.

Step 2: Visualize (Gemini 2.5 Flash Image)

The agent has successfully found a list of products. Now, the next step in our workflow is to bridge the gap between a list of items and a cohesive look. How do these products look together? And more importantly, how could we create a photorealistic visualization of them on the user?

This is where Gemini 2.5 Flash Image (Nano Banana) steps in. Here’s how this step works in practice:

1. The user selects their favorite items from the search results (e.g., one blazer, one shirt, one pair of pants, one pair of shoes).
2. They upload a photo of themselves (the “original asset”).
3. Our application creates a simple product collage from the selected items. This is a crucial step because Gemini 2.5 Flash Image (Nano Banana) accepts a maximum of three input images. To send the user’s photo plus a full outfit of 3–4 items, we must first combine all the product images into a single collage.
4. We send the user’s photo and the product collage to the Gemini 2.5 Flash Image model with a clear, simple prompt.

# From vto_generator.py - Nano Banana Virtual Try-On
def generate_multi_product_tryon(user_photo_base64, products, gemini_client):
    # ... (Logic to decode user photo)
    # ... (Logic to download product images)
    # 1. Create the collage (if more than 2 items)
    collage_bytes = create_product_collage(product_images_bytes)
    
    # 2. Build the prompt
    prompt = "Generate a picture of the person in the first image wearing all the clothes shown in the collage photo (second image)."
    
    model = "gemini-2.5-flash-image"
    
    # 3. Build the content parts
    parts = [
        types.Part.from_text(text=prompt),
        types.Part.from_bytes(data=user_photo_bytes, mime_type="image/jpeg"), # Image 1
        types.Part.from_bytes(data=collage_bytes, mime_type="image/jpeg")      # Image 2
    ]
    contents = [types.Content(role="user", parts=parts)]
    # 4. Generate the new image
    response = gemini_client.models.generate_content_stream(
        model=model,
        contents=contents,
        # Omitted for brevity
    )
    
    # ... (Extract image from response)

In seconds, the user receives a single, photorealistic Virtual Try-On (VTO) image of themselves wearing the complete outfit they just assembled from the agent’s search results.

Step 3: Animate (Veo 3.1)

We have a static, photorealistic image, which is a fantastic result. But to truly enhance the user experience, we can take it one step further. A dynamic video gives the user a complete, 360-degree view, helping them understand the fit and flow of the outfit in a way a static image can’t. The final step in our workflow is to bring this static image to life

This final animation step is powered by Veo 3.1, Google’s most capable video generation model. Veo can take a starting image and a text prompt to generate a high-definition, coherent video with complex and realistic camera movements.

Our workflow logic for this final step is:

1. We take the static virtual try-on image generated by Nano Banana in Step 2. This will be the starting frame for our video.
2. Our application logic checks the list of selected items. If it detects “shoes,” “boots,” or other footwear, it knows a full-body shot is required.
3. Based on this check, it builds an intelligent prompt:
4. If shoes are present: It requests a “smooth 360-degree camera rotation” to show the full outfit from head to toe.
5. If no shoes are present: It requests a “smooth 180-degree rotation” to focus on the upper body.
6. This starting image and the custom prompt are sent to the Veo 3.1 API.

This “smart prompt” logic ensures the generated video is always framed correctly for the specific outfit.

# From video_generator.py - Veo 3.1 Video Generation

def generate_outfit_video_veo(user_photo_base64, products, project_id, ...):
    
    # ... (Logic to check for shoes)
    has_shoes = any(keyword in name.lower() for name in product_names ...)
    
    # 1. Select the right camera movement based on presence of shoes
    if has_shoes:
        # Full outfit with shoes - 360° rotation
        prompt = f"""A professional video showing the full outfit.
SUBJECT: The person wearing {len(products)} new clothing items.
CAMERA MOVEMENT: Smooth 360-degree clockwise rotation around the person, showing all angles.
VISUAL STYLE: Full body shot (head to toe visible), well-lit, simple neutral background. Keep the person centered.
"""
    else:
        # Upper body outfit - 180° rotation
        prompt = f"""A professional video showing the upper-body outfit.
SUBJECT: The person wearing {len(products)} new clothing items.
CAMERA MOVEMENT: Smooth 180-degree rotation (front to back to front).
VISUAL STYLE: Medium shot focusing on the upper body, well-lit, simple neutral background. Keep the person centered.
"""
    
    # 2. Load the try-on image (output from Step 2)
    tryon_image = types.Image.from_file(location=temp_path)
    
    # 3. Call the Veo 3.1 API
    operation = client.models.generate_videos(
        model="veo-3.1-generate-preview",
        prompt=prompt,
        image=tryon_image, # Use the try-on as the starting frame
        config=types.GenerateVideosConfig(
            aspect_ratio="9:16", # Vertical for outfit showcase
            duration_seconds=8,
            resolution="1080p",
        ),
    )
    
    # ... (Wait for operation to complete)

The result is a 8-second, high-definition, vertical video of the user in their new outfit, complete with smooth, professional camera motion. It’s a dynamic asset that gives a 360° view of the products, taking the shopping experience to a whole new level.

Beyond Apparel: The Workflow’s Versatility

While we used outfit generation as the main example, this agentic workflow generalizes beautifully to many domains. Whether it’s home decor, event planning, or recipe creation, the pattern Decompose → Search → Visualize → Animate remains the same.

Example 1: Home Decor

• Prompt: “I need a mid-century modern look for my living room under $2000.”
• Step 1: The agent browses sites like Wayfair or IKEA, searching for “mid-century sofa,” “walnut coffee table,” and “abstract rug” that fit the budget. It gathers a list of products.
• Step 2: The user uploads a photo of their empty living room...
• Step 3: The model takes the “virtually staged” room and generates a short video, smoothly panning across the room to showcase the new furniture from different angles.

Example 2: Professional Recipe Showcase

• Prompt: “I need a 3-course Italian dinner menu for a corporate event this week.”
• Step 1: The agent decomposes this into “Italian appetizer,” “Italian main course,” and “Italian dessert.” It browses recipe sites to find 3–4 top-rated options for each, extracting ingredients and instructions.
• Step 2: The user selects one recipe from each category. They upload a photo of a simple dinner plate. Nano Banana generates a photorealistic image of the final, plated meal.
• Step 3: The model takes the plated meal image and generates a short, appetizing promotional video, perhaps with a slow zoom or a subtle steam effect, making the recipe irresistible.

Demo: AI Shopping Workflow in Action

Watch the end-to-end workflow in action, from intent understanding to product search, virtual try-on, and video generation.

👉 Note: For demonstration purposes, some browsing sequences in the video have been accelerated. In real-world usage, the complete workflow (searching 4 items across Amazon.fr, generating virtual try-on, and creating video) takes approximately 8–9 minutes from start to finish.

Conclusion

What we’ve built here is just one of many possible generative AI workflows. It shows the power of specialized multimodal models accessible via APIs, each designed for a specific task, from reasoning and computer use to image and video generation.

By combining these capabilities, we can build complex, end-to-end applications that move beyond simple Q&A to actively assist users in completing real-world tasks. As models become more multimodal and compositional, these orchestrated pipelines are shaping the next generation of AI experiences, ones that understand, act, and collaborate with users seamlessly.

Responsible Use

Gemini 2.5 Computer Use was built with safety at its core. Developers should follow the best practices outlined in the documentation, including per-step safety checks, user confirmation for sensitive actions, and thorough testing before deployment. Use these capabilities responsibly, with transparency, respect for website policies, and a focus on trust, safety, and human benefit.

Disclaimer

This project is a technical proof-of-concept for educational purposes. The workflow shown in the demo is intended to illustrate the capabilities of these AI models. The use of a commercial website and its content is for demonstration purposes only.

A real-world application of this technology would require explicit permission from website owners to ensure it aligns with their policies and content usage guidelines.

📜About the author

Dr. Wafae Bakkali is a Staff Generative AI Specialist, Blackbelt at Google. With over 13 years of experience, she has a proven track record of combining deep technical expertise and leadership with strategic business insights to drive impactful outcomes. Through her current role at Google and former roles at reputable organizations such as AWS, Dr. Bakkali has successfully guided organizations across various industries in adopting and building AI solutions.

She holds a PhD from IMT Atlantique, France, and served as a postdoctoral researcher at Centrale Supélec, Paris-Saclay University, France.

Dr. Bakkali is also an active contributor to the AI community. She regularly shares her expertise as a speaker at international tech conferences and mentors participants in AI hackathons, tech, and startup events.

🔗Connect with me on LinkedIn: https://www.linkedin.com/in/wafae-bakkali/

Building a Multimodal AI Shopping Workflow with Gemini 2.5 was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

👉 The complete code for this analysis is available on GitHub.

Introduction

In multimodal generative AI, precise control over image attributes such as color, texture, and composition is essential for producing professional, brand-consistent content. Among these attributes, color plays a particularly critical role. Image-generative models can approximate colors well, but exact pixel-perfect adherence to a specific HEX, RGB or HSL code is still a challenge. Even small discrepancies between a specified color value and the final color rendered in the generated image can disrupt workflows and compromise brand identity.

This level of accuracy is vital for applications such as on-brand marketing materials, digital art, e-commerce imagery, social media assets, product mockups, user interface elements or web design. While minor color variations may be acceptable for casual use, professional settings require strict adherence to predefined color palettes.

In this post:

• We show how to achieve precise color code matching in AI-generated images using visual references.
• We provide examples with Gemini 2.5 Flash Image (aka “Nano Banana”) and evaluate how well the model reproduces target colors specified in HEX, RGB and HSL formats. For that, we analyze 50 generated images created with the model.
• For objective evaluation, color accuracy is measured using Delta E 2000 (CIE2000), an industry-standard metric for perceptual color difference

The “Color Image Reference” Approach

Typically, users specify a color in the text prompt (e.g., “a car in color #FF0000” or “a car in RGB(255,0,0)”). While this can work in some cases, it is often inaccurate because the model does not interpret the instruction with mathematical precision. This can result in noticeable discrepancies between the prompt and the generated image.

The solution? Instead of telling the model the color, show it.

This is a two-step process:

1. Convert: Turn your color code (e.g., HEX) into a simple, flat image swatch.
2. Reference: Pass this image swatch as a multimodal input along with your text prompt.

Step 1: convert a color code to a swatch

We can use the Python Pillow (PIL) library to create an in-memory PNG of any color. The following function takes a hex code, converts it to RGB, and saves it as bytes, ready to be sent to the API:

def hex_to_rgb(h: str) -> tuple:
    """Converts a hex color string (e.g. #FF0000) to an (R, G, B) tuple."""
    h = h.lstrip("#")
    return tuple(int(h[i : i + 2], 16) for i in (0, 2, 4))
def create_color_swatch_bytes(hex_color: str) -> bytes:
    """Creates a 256x256 PNG image of a solid color in memory."""
    # Convert the hex code to a numerical (R, G, B) tuple
    rgb_color = hex_to_rgb(hex_color)
    
    # Create a new image in 'RGB' mode with the specified color
    img = Image.new("RGB", (256, 256), color=rgb_color)
    
    # Save the image to an in-memory byte buffer
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    
    # Return the raw bytes of the image
    return buf.getvalue()
# --- Example Usage ---
hex_code = "#d81515"
image_bytes = create_color_swatch_bytes(hex_code)
image_swatch = Image.open(io.BytesIO(image_bytes))

Step 2: use the swatch in the Gemini prompt

With our image_swatch object, we now build a multimodal prompt. The contents list will include our base prompt (defining the object) and the image_swatch (defining the color), as shown in the example below:

# Assuming 'client' is the genai module and the model is defined
# We pass the PIL.Image object directly in the contents list
parts_image = [
    f"{base_prompt} The color should match the attached swatch.",
    image_swatch  
]
# This list is sent to the model:
client.models.generate_content(contents=parts_image)

👉 For the complete, runnable code, including client setup and helper functions, see the full repository [linked in the Appendix].

Evaluation

In this section, we conduct a quantitative evaluation to assess how well Gemini 2.5 Flash Image adheres to specific color codes and compare the results of prompting the model with color text-only descriptions versus text with color image references.

• Text-only prompt:
  [f"{base_prompt} The color is {hex_code}."]
• Color image reference prompt:
  [f"{base_prompt} The color should match the attached swatch.", image_swatch]

We performed this test using 50 randomly selected, diverse hex color codes to ensure a representative sample across the color spectrum. For each prompt type (text-only and text with a color image reference), Gemini generated an image, from which we computed the average color of the output.

We then measured the perceptual color difference using ΔE2000, comparing the average generated color to the exact target color code.

To ensure the outputs were suitable for this kind of evaluation, we included the following instruction in the prompt:

“A full-frame, solid-color background. The entire image should be a single, flat, unified color. No texture, no lighting, no gradients.”

This ensured that the generated images were visually flat and uniform, allowing for accurate color averaging and meaningful Delta E 2000 (ΔE2000) comparisons.

Example Comparison

As shown in Figure 1, the results of the two prompting strategies for the target color #d81515 are compared:

• Target: A flat red with hex code #d81515
• Text-only color instruction result: ΔE = 9.46 (visibly darker)
• Color image reference result: ΔE = 0.66 (nearly indistinguishable from the target)

Results across all 50 tests

The results across all 50 different color tests show a significant improvement in color reproduction accuracy with the color image reference approach:

• Average ΔE (Text-only): 11.25
• Average ΔE (Image reference): 0.90

A ΔE below 1.0 is generally considered imperceptible to the human eye, indicating that the color image-based approach enables high-fidelity color reproduction, with the average error (0.90) falling below the perceptible threshold, while a higher value indicates greater perceptual deviation, as seen with text-only descriptions, which lead to larger color errors.

To illustrate these findings, Figure 2 shows the distribution of ΔE values across all 50 test cases.

Additional Examples

To further illustrate the performance across different tasks, we include additional examples that highlight the real-world implications of using text-only versus image-reference instructions. These examples cover both image generation and image editing tasks.

Image generation examples

• Figure 3 shows an image generation result using a text-only prompt.
• Figure 4 presents the same generation task, but with an image swatch provided as a reference, allowing the model to match the color more precisely.

Image editing examples

• Figure 5 illustrates an image editing task where color modification is prompted through text alone.
• Figure 6 shows the corresponding image editing task using a color image reference, enabling more accurate and faithful edits.

Conclusion

In this post, we explored how to achieve precise color reproduction with AI image generation models, using Gemini 2.5 Flash Image as an example. We compared the effectiveness of text-only prompts versus color image reference-based generation. The results clearly show that the color image reference method leads to near-perfect color reproduction, with an average Delta E score of 0.90, a value generally considered imperceptible to the human eye. This highlights Gemini’s ability to match colors with remarkable precision, ensuring high-fidelity color replication across the entire color spectrum.

When it comes to selecting the right approach for achieving accurate color reproduction, text-only prompts may be suitable for quick, creative generation tasks. However, for more professional projects or scenarios requiring brand-accurate color replication, leveraging a color image reference with Gemini 2.5 Flash Image can unlock the model’s full potential, delivering precise color matching and ensuring high-quality, accurate color results.

Appendix

1. For complete results, further exploration, or reproducing the results, see the complete code used for this blog on GitHub.
2. For those interested in further exploring Gemini 2.5 Flash Image and its capabilities, you can try the model on Google AI Studio or refer to the official documentation for deeper insights.

📜About the author

Dr. Wafae Bakkali is a Staff Generative AI Specialist, Blackbelt at Google. With over 13 years of experience, she has a proven track record of combining deep technical expertise and leadership with strategic business insights to drive impactful outcomes. Through her current role at Google and former roles at reputable organizations such as AWS, Dr. Bakkali has successfully guided organizations across various industries in adopting and building AI solutions.

She holds a PhD from IMT Atlantique, France, and served as a postdoctoral researcher at Centrale Supélec, Paris-Saclay University, France.

Dr. Bakkali is also an active contributor to the AI community. She regularly shares her expertise as a speaker at international tech conferences and mentors participants in AI hackathons, tech, and startup events.

🔗Connect with me on LinkedIn: https://www.linkedin.com/in/wafae-bakkali/

Color Code Adherence in AI Image Generation: A Case Study with Gemini was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Veo 3 is Google’s most advanced Artificial Intelligence (AI) video generation model to date. It creates high-quality videos from text or images and can even generate synchronized audio from a single prompt. As one of the most powerful generative video models available today, Veo 3 unlocks new creative possibilities for filmmakers, marketers, and content creators.

Creating compelling videos with generative AI requires thoughtful prompting. This requires a different approach than prompting a large language model.

In this guide:

• We share experimentation-based prompting guidance for Veo 3.
• You will learn how to structure effective prompts for Veo 3, apply best practices for video generation and avoid common pitfalls even without prior experience in AI or video production.

The Anatomy of a Veo 3 Prompt

Creating a great Veo 3 prompt is different from prompting a chatbot. It’s about using the right terminology and prompt structure. Breaking your creative idea down into key components is the most effective way to guide Veo 3 toward your desired outcome.

Figure 1, below, illustrates this structure, outlining the nine core elements of an expert-level prompt. These elements represent the key building blocks you may use when crafting your prompt. However, it’s important to note that you don’t need to include all of them in every prompt. The right combination depends on your specific goals.

In the following sections, we will break down each of the nine elements in detail, sharing supported options and examples to help you apply them effectively in your Veo 3 prompts.

1- Subject

Common options for the subject include, but are not limited to:

• People: Man, woman, elderly person, specific professions (e.g., “a seasoned detective”, “a joyful baker”, “a futuristic astronaut”), historical figures, mythical beings (e.g., “a mischievous fairy”, “a stoic knight”).
• Animals: Specific breeds (e.g., “a playful Golden Retriever puppy”, “a majestic bald eagle”, “a sleek black panther”), fantastical creatures (e.g., “a miniature dragon with iridescent scales”, “a wise, ancient talking tree”).
• Objects: Everyday items (e.g., “a vintage typewriter”, “a steaming cup of coffee”, “a worn leather-bound book”), vehicles (e.g., “a classic 1960s muscle car”, “a futuristic hovercraft”, “a weathered pirate ship”), abstract shapes (“glowing orbs”, “crystalline structures”).
• Multiple Subjects: You can combine people, animals, objects, or any mix of them in the same video (e.g., “A group of diverse friends laughing around a campfire while a curious fox watches from the shadows”, “a busy marketplace scene with vendors and shoppers”).

Example: complex details with multiple subjects

2- Action

Actions can include, but are not limited to, the following examples:

• Basic Movements: e.g., Walking, running, jumping, flying, swimming, dancing, spinning, falling, standing still, sitting.
• Interactions: e.g., Talking, laughing, arguing, hugging, fighting, playing a game, cooking, building, writing, reading, observing.
• Emotional Expressions: e.g., Smiling, frowning, looking surprised, concentrating deeply, appearing thoughtful, showing excitement, crying.
• Subtle Actions: e.g., A gentle breeze ruffling hair, leaves rustling, a subtle nod, fingers tapping impatiently, eyes blinking slowly.
• Transformations/Processes: e.g., A flower blooming in fast-motion, ice melting, a city skyline developing over time (though keep clip length in mind).

Example: Directing a story by sequencing actions and emotional changes

3- Scene / context

The most common options for the scene include:

• Location (Interior): e.g., A cozy living room with a crackling fireplace, a sterile futuristic laboratory, a cluttered artist’s studio, a grand ballroom, a dusty attic.
• Location (Exterior): e.g., A sun-drenched tropical beach, a misty ancient forest, a bustling futuristic cityscape at night, a serene mountain peak at dawn, a desolate alien planet.
• Time of Day: e.g., Golden hour, midday sun, twilight, deep night, pre-dawn.
• Weather: e.g., Clear blue sky, overcast and gloomy, light drizzle, heavy thunderstorm with visible lightning, gentle snowfall, swirling fog.
• Historical/Fantastical Period: e.g., A medieval castle courtyard, a roaring 1920s jazz club, a cyberpunk alleyway, an enchanted forest glade.
• Atmospheric Details: e.g., Floating dust motes in a sunbeam, shimmering heat haze, reflections on wet pavement, leaves scattered by the wind.

Example: Building an immersive world

4- Camera angles

Here are some of the most common camera angles and the effects they typically create:

• Eye-Level Shot: Offers a neutral, common perspective, as if viewed from human height. “Eye-level shot of a woman sipping tea.”
• Low-Angle Shot: Positions the camera below the subject, looking up, making the subject appear powerful or imposing. “Low-angle tracking shot of a superhero landing.”
• High-Angle Shot: Places the camera above the subject, looking down, which can make the subject seem small, vulnerable, or part of a larger pattern. “High-angle shot of a child lost in a crowd.”
• Bird’s-Eye View / Top-Down Shot: A shot taken directly from above, offering a map-like perspective of the scene. “Bird’s-eye view of a bustling city intersection.”
• Worm’s-Eye View: A very low-angle shot looking straight up from the ground, emphasizing height and grandeur. “Worm’s-eye view of towering skyscrapers.”
• Dutch Angle / Canted Angle: The camera is tilted to one side, creating a skewed horizon line, often used to convey unease, disorientation, or dynamism. “Dutch angle shot of a character running down a hallway.”
• Close-Up: Frames a subject tightly, typically focusing on a face to emphasize emotions or a specific detail. “Close-up of a character’s determined eyes.”
• Extreme Close-Up: Isolates a very small detail of the subject, such as an eye or a drop of water. “Extreme close-up of a drop of water landing on a leaf.”
• Medium Shot: Shows the subject from approximately the waist up, balancing detail with some environmental context, common for dialogue. “Medium shot of two people conversing.”
• Full Shot / Long Shot: Shows the entire subject from head to toe, with some of the surrounding environment visible. “Full shot of a dancer performing.”
• Wide Shot / Establishing Shot: Shows the subject within their broad environment, often used to establish location and context at the beginning of a sequence. “Wide shot of a lone cabin in a snowy landscape.”
• Over-the-Shoulder Shot: Frames the shot from behind one person, looking over their shoulder at another person or object, common in conversations. “Over-the-shoulder shot during a tense negotiation.”
• Point-of-View Shot: Shows the scene from the direct visual perspective of a character, as if the audience is seeing through their eyes. “POV shot as someone rides a rollercoaster.”

Example: Bird’s-eye view

Example: Extreme close-up

5- Camera movements

Here are some of the most common camera movements you can use in your prompts:

• Static Shot (or fixed): The camera remains completely still; there is no movement. “Static shot of a serene landscape.”
• Pan (left/right): The camera rotates horizontally left or right from a fixed position. “Slow pan left across a city skyline at dusk.”
• Tilt (up/down): The camera rotates vertically up or down from a fixed position. “Tilt down from the character’s shocked face to the revealing letter in their hands.”
• Dolly (In/Out): The camera physically moves closer to the subject or further away. “Dolly out from the character to emphasize their isolation.”
• Truck (Left/Right): The camera physically moves horizontally (sideways) left or right, often parallel to the subject or scene. “Truck right, following a character as they walk along a busy sidewalk.”
• Pedestal (Up/Down): The camera physically moves vertically up or down while maintaining a level perspective. “Pedestal up to reveal the full height of an ancient, towering tree.”
• Zoom (In/Out): The camera’s lens changes its focal length to magnify or de-magnify the subject. This is different from a dolly, as the camera itself does not move. “Slow zoom in on a mysterious artifact on a table.”
• Crane Shot: The camera is mounted on a crane and moves vertically (up or down) or in sweeping arcs, often used for dramatic reveals or high-angle perspectives. “Crane shot revealing a vast medieval battlefield.”
• Aerial Shot / Drone Shot: A shot taken from a high altitude, typically using an aircraft or drone, often involving smooth, flying movements. “Sweeping aerial drone shot flying over a tropical island chain.”
• Handheld / Shaky Cam: The camera is held by the operator, resulting in less stable, often jerky movements that can convey realism, immediacy, or unease. “Handheld camera shot during a chaotic marketplace chase.”
• Whip Pan: An extremely fast pan that blurs the image, often used as a transition or to convey rapid movement or disorientation. “Whip pan from one arguing character to another.”
• Arc Shot: The camera moves in a circular or semi-circular path around the subject. “Arc shot around a couple embracing in the rain.”

Example: Zoom-in shot

Example: Aerial drone shot

6- Lens and optical effects

Here are some common lenses and optical effects you can use to enhance your video:

• Wide-Angle Lens (e.g., “18mm lens,” “24mm lens”): Captures a broader field of view than a standard lens. It can exaggerate perspective, making foreground elements appear larger and creating a sense of grand scale or, at closer distances, distortion. “Wide-angle lens shot of a grand cathedral interior, emphasizing its soaring arches.”
• Telephoto Lens (e.g., “85mm lens,” “200mm lens”): Narrows the field of view and compresses perspective, making distant subjects appear closer and often isolating the subject by creating a shallow depth of field. “Telephoto lens shot capturing a distant eagle in flight against a mountain range.”
• Shallow Depth of Field / Bokeh: An optical effect where only a narrow plane of the image is in sharp focus, while the foreground and/or background are blurred. The aesthetic quality of this blur is known as ‘bokeh’. “Portrait of a man with a shallow depth of field, their face sharp against a softly blurred park background with beautiful bokeh.”
• Deep Depth of Field: Keeps most or all of the image, from foreground to background, in sharp focus. “Landscape scene with deep depth of field, showing sharp detail from the wildflowers in the immediate foreground to the distant mountains.”
• Lens Flare: An effect created when a bright light source directly strikes the camera lens, causing streaks, starbursts, or circles of light to appear in the image. Often used for dramatic or cinematic effect. “Cinematic lens flare as the sun dips below the horizon behind a silhouetted couple.”
• Rack Focus: The technique of shifting the focus of the lens from one subject or plane of depth to another within a single, continuous shot. “Rack focus from a character’s thoughtful face in the foreground to a significant photograph on the wall behind them.”
• Fisheye Lens Effect: An ultra-wide-angle lens that produces extreme barrel distortion, creating a circular or strongly convex, wide panoramic image. “Fisheye lens view from inside a car, capturing the driver and the entire curved dashboard and windscreen.”
• Vertigo Effect (Dolly Zoom): A camera effect achieved by dollying the camera towards or away from a subject while simultaneously zooming the lens in the opposite direction. This keeps the subject roughly the same size in the frame, but the background perspective changes dramatically, often conveying disorientation or unease. “Vertigo effect (dolly zoom) on a character standing at the edge of a cliff, the background rushing away.

Example: Shallow Depth of Field

Example: Rack focus shot

7- Visual style & aesthetics

This broad category can be broken down into four key components:

7.1. Lighting:

• Natural Light: “Soft morning sunlight streaming through a window,” “Overcast daylight”, “Moonlight.”
• Artificial Light: “Warm glow of a fireplace”, “Flickering candlelight,” “Harsh fluorescent office lighting”, “Pulsating neon signs.”
• Cinematic Lighting: “Rembrandt lighting on a portrait”, “Film noir style with deep shadows and stark highlights”, “High-key lighting for a bright, cheerful scene”, “Low-key lighting for a dark, mysterious mood.”
• Specific Effects: “Volumetric lighting creating visible light rays”, “Backlighting to create a silhouette”, “Golden hour glow”, “Dramatic side lighting.”

7.2. Tone / Mood

• Happy/Joyful: Bright, vibrant, cheerful, uplifting, whimsical.
• Sad/Melancholy: Somber, muted colors, slow pace, poignant, wistful.
• Suspenseful/Tense: Dark, shadowy, quick cuts (if implying edit), sense of unease, thrilling.
• Peaceful/Serene: Calm, tranquil, soft, gentle, meditative.
• Epic/Grandiose: Sweeping, majestic, dramatic, awe-inspiring.
• Futuristic/Sci-Fi: Sleek, metallic, neon, technological, dystopian, utopian.
• Vintage/Retro: Sepia tone, grainy film, specific era aesthetics (e.g., “1950s Americana”, “1980s vaporwave”).
• Romantic: Soft focus, warm colors, intimate.
• Horror: Dark, unsettling, eerie, gory (though be mindful of content filters).

7.3. Artistic style

• Photorealistic: “Ultra-realistic rendering”, “Shot on 8K camera.”
• Cinematic: “Cinematic film look”, “Shot on 35mm film”, “Anamorphic widescreen.”
• Animation Styles: “Japanese anime style”, “Classic Disney animation style,” “Pixar-like 3D animation”, “Claymation style”, “Stop-motion animation”, “Cel-shaded animation.”
• Art Movements/Artists: “In the style of Van Gogh”, “Surrealist painting,” “Impressionistic”, “Art Deco design”, “Bauhaus aesthetic.”
• Specific Looks: “Gritty graphic novel illustration”, “Watercolor painting coming to life”, “Charcoal sketch animation”, “Blueprint schematic style.

7.4. Ambiance

• Color Palettes: “Monochromatic black and white”, “Vibrant and saturated tropical colors”, “Muted earthy tones”, “Cool blue and silver futuristic palette”, “Warm autumnal oranges and browns.”
• Atmospheric Effects: “Thick fog rolling across a moor”, “Swirling desert sands”, “Gentle falling snow creating a soft blanket”, “Heat haze shimmering above asphalt”, “Magical glowing particles in the air”, “Subsurface scattering on a translucent object.”
• Textural Qualities: “Rough-hewn stone walls”, “Smooth, polished chrome surfaces”, “Soft, velvety fabric”, “Dewdrops clinging to a spiderweb.”

Example: Animation style

Example: Vintage style

8- Temporal elements

Common temporal elements you can specify include:

• Pacing: “Slow-motion”, “Fast-paced action”, “Time-lapse”.
• Evolution (subtle for short clips): “A flower bud slowly unfurling”, “A candle burning down slightly”, “Dawn breaking, the sky gradually lightening.”
• Rhythm: “Pulsating light”, “Rhythmic movement.”

Example: Evolution

Example: Time-lapse

9- Audio

Common audio elements you can describe in your prompt include:

• Sound Effects: Individual, distinct sounds that occur within the scene (e.g., “the sound of a phone ringing”, “water splashing in the background”, “soft house sounds, the creak of a closet door, and a ticking clock” ).
• Ambient Noise: The general background noise that makes a location feel real (e.g., “the sounds of city traffic and distant sirens”, “waves crashing on the shore”, “the quiet hum of an office” ).
• Dialogue: Spoken words from characters or a narrator (e.g., “The man in the red hat says: ‘Where is the rabbit?’”, “A voiceover with a polished British accent speaks in a serious, urgent tone”, “Two people discuss a movie” ).

Example: Dialogue

Tips and best practices

Beyond understanding the individual components, here are some best practices to help you combine them effectively and refine your results.

1- Use advanced cinematic language

While we’ve covered basic camera moves, you can suggest more advanced cinematic language in your prompts. Try incorporating terms for editing styles and specific techniques. Consider using terms like: “match cut”, “jump cut”, “establishing shot sequence”, “montage (of brief, related actions)”, “split diopter effect”.

Example: Jump cut

2- Use clear and specific language

For instance, instead of a confusing description like the one below:

• Potentially confusing: “I envision a scene where, like, the main focus, a dude, is kinda sad, and it’s like, dark, and the camera is sort of, from below, you know?”

Provide a clear and direct prompt that eliminates ambiguity:

• Clear and direct: “Low-angle close-up shot of a man with a somber expression. The scene is dimly lit, conveying a melancholic mood.”

3- Pro tip for dialogue: How to avoid unwanted subtitles

To ensure your dialogue is spoken by the character without appearing as subtitles, use only a colon (:) after the speaker's action. Avoid using quotation marks (" "), as the model may interpret them as text to display visually.

• DO: A women says: My name is Clara
• Don’t: A women says: “My name is Clara”

4- Maximize impact with multiple orientations

To ensure your video performs effectively on all platforms, consider generating it in multiple aspect ratios. Different platforms are optimized for different shapes, and creating versions for each is especially critical for professional use cases like marketing and advertising.

Here are the key aspect ratios and their primary uses:

• 16:9 (Landscape): The standard for YouTube, presentations, and most traditional video displays.
• 9:16 (Portrait/Vertical): Essential for mobile-first platforms like TikTok, Instagram Reels, and YouTube Shorts.

5- Focus on a single scene

For the best results, dedicate each prompt to a single, focused moment. Trying to chain multiple distinct events (A then B then C) in one prompt for a short clip often leads to muddled or incomplete videos.

Instead of one complex prompt:

“A detective finds a clue in a library, then drives across the city at night, and then confronts a suspect in a warehouse.”

Generate each part as a separate clip:

• Clip 1: “Close-up on a detective’s gloved hand brushing dust off an old book in a dark library, revealing a hidden symbol.”
• Clip 2: “A car driving through a neon-lit city at night, with rain streaking across the windshield, in a film noir style.”
• Clip 3: “Inside a shadowy warehouse, a detective stands opposite a silhouetted figure, creating a tense atmosphere.”

6- Enhance Your Workflow with Gemini

Gemini can be a powerful partner throughout your entire video creation process, from ideation to evaluation.

Before Creation: Use Gemini as an expert prompter

Instead of starting from scratch, you can ask Gemini to act as an expert prompter. Have it refine your basic ideas into detailed, veo-ready prompts. You can give it an instruction like this:

Example prompt for Gemini: “Act as an expert prompter for a generative AI video generation model. Look at this image, and write a prompt that <instruction>. Ensure your prompt is comprehensive and detailed.”

After Creation: Use Gemini as a “Second pair of eyes”

Once your video is generated, Gemini can evaluate the final output, check it against company or brand guidelines, and flag any potentially problematic areas that may require human review.

7- (Image to video) Use a high-quality source image

When using the image-to-video feature, the quality of your source image is paramount. Veo uses this image as the foundational blueprint for everything that follows, including character detail, lighting, and overall artistic style.

A sharp, clear, and well-composed photograph will always yield a more coherent and higher-quality video. Think of your source image as the first frame of your film, the stronger the start, the better the finish.

8- (Image to video) Prompt for motion only

This is the most important principle of Image-to-Video. Your source image already provides the subject, scene, and style. Your text prompt should only describe the motion you want to see.

• DO: Prompt for camera movement, subject animation, and environmental changes.
• DON’T: Re-describe the character, the background, or the lighting. Redundant prompts confuse the model and lead to poor results.

Pro Tip: In your motion prompt, refer to the character with general terms like “the subject”, “the woman” or “he/she.”

9- (Image to video) Three ways to animate your image

You can direct three types of movement, either alone or in combination.

Camera Motion: The camera moves, but the scene is static. This is the simplest and most reliable way to add dynamism.

• Example: “Slow dolly in on the subject.”

Subject Animation: The main character or object moves. Best for subtle, lifelike actions.

• Example: “The character’s hair and clothes flutter gently in the wind.”

Environmental Animation: The background or atmosphere comes alive.

• Example: “Fog rolls in slowly across the landscape.”

Example: Subject animation

10- How to achieve character and voice consistency

When creating long videos, character and voice consistency is key. Follow these steps to maintain the best level of coherence throughout your scenes:

• Create a detailed character description: Your description is the foundation for consistency. To ensure reusability and voice consistency, build out the description for your character with a rich set of unchangeable features: physical build and age, hair color and style, facial structure, eye color and shape, any defining marks and a voice style. You can use Gemini to generate an exhaustive verbal description of your character’s facial features.
• Apply the description consistently: Copy and paste the entire, unchanged character description into your prompt for every new scene. Only modify the parts that describe the new action or setting. To speed up your workflow, you can also use Gemini as a scene generator. Provide it with your final character description and ask it to generate multiple scene prompts for you.
• Use the same seed parameter: Seed numbers are numerical values that control the randomness in the AI’s generation process. Using the same seed with the same prompt will likely generate very similar outputs, while altering the seed can lead to variations. For specific details on how to integrate the seed parameter, refer to the VEO API specification.

Example: Character and voice consistency

Conclusion

This guide shares experimentation-based best practices for structuring prompts, helping you create compelling videos with Veo 3 and bring your creative concepts to life.

As you create, keep in mind that Veo applies safety filters across Vertex AI to help ensure that generated videos and uploaded photos don’t contain offensive content. For example, prompts that violate responsible AI guidelines are blocked.

Acknowledgments

The author would like to thank G. Hussain Chinoy for their earlier work that inspired this post, and feedback on this guidance.

📜About the author

Dr. Wafae Bakkali is a Staff Generative AI Specialist, Blackbelt at Google. With over 12 years of experience, she has a proven track record of combining deep technical expertise and leadership with strategic business insights to drive impactful outcomes. Through her current role at Google and former roles at reputable organizations such as AWS, Dr. Bakkali has successfully guided organizations across various industries in adopting and building AI solutions.

She holds a PhD from IMT Atlantique, France, and served as a postdoctoral researcher at Centrale Supélec, Paris-Saclay University, France.

Dr. Bakkali is also an active contributor to the AI community. She regularly shares her expertise as a speaker at international tech conferences and mentors participants in AI hackathons, tech, and startup events.

🔗Connect with me on LinkedIn: https://www.linkedin.com/in/wafae-bakkali/

Veo 3: A Detailed Prompting Guide was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Generative AI is one of the most transformative, rapidly evolving, and widely adopted technologies today. A key reason for its impact is its ability to enable humans to communicate with computers using natural language, making interactions easy and intuitive. This ability has facilitated its adoption in our daily lives and across multiple industries, spanning sectors from education and healthcare to entertainment.

The way we interact started with text, powered by large language models (LLMs). Now, to further bridge the gap between human and machine communication, the natural evolution is moving beyond writing to speaking. This is driving a growing focus in research on multimodal generative AI, which refers to systems capable of processing and generating multiple forms of data (modalities), such as text, speech, and images.

In this post, we focus on speech-to-speech models, which are generative AI models designed to process spoken input and respond in real-time with natural, human-like speech, opening new possibilities for technological innovation and applications.

Here’s what we’ll explore:

• Under the Hood: Key components of speech-to-speech generative AI models.
• Real-World Applications: Use cases, capabilities and best practices for building with speech-to-speech generative AI models.
• Measuring Success: Essential evaluation metrics for these systems.

Overview

Given the wide availability of LLMs trained on massive text datasets and the challenges associated with collecting and annotating large amounts of speech data, the natural approach to building speech-to-speech generative AI systems has been to leverage these LLMs and adapt them for spoken interactions. Broadly speaking, this adaptation of LLMs for speech has been done in two main approaches: using a pipeline of cascaded independent modules or employing an end-to-end speech large language model that directly handles the speech modality. In this post, we will refer to these approaches as cascaded and end-to-end speech-to-speech systems. Figure 1 illustrates these two approaches.

📜 N.B: The discussion throughout this post focuses on decoder-only (autoregressive) transformer architectures, as they are representative of the majority of current LLMs and speech-to-speech models.

1- Cascaded speech-to-speech systems

Cascaded systems operate as a pipeline, with processing handled by independent modules. The main components include Automatic Speech Recognition (ASR) to transcribe the spoken input into text, an LLM to process and generate text responses, and Text-to-Speech (TTS) to convert the generated text back into speech. AudioGPT is an example of a cascaded speech-to-speech system.

Despite its modularity, the reliance on separate modules in cascaded systems can introduce latency making real-time conversation challenging. Additionally, relying on intermediate text representations can sometimes lose nuances in the original speech, such as emotion or tone.

2- End-to-end speech-to-speech models

In contrast, end-to-end speech-to-speech models are capable of handling the speech modality by directly extracting information from speech inputs and generating speech representations without relying on intermediate text representation. End-to-end systems offer a more integrated approach, often resulting in lower latency and a more natural conversational flow.

It’s worth noting that there are also hybrid approaches, where some cascaded systems skip the ASR module by handling input speech directly but still need the TTS module, such as Qwen2-Audio or VITA. The choice between cascaded and end-to-end systems depend on the specific needs and constraints of the real-world application.

As current research primarily focuses on end-to-end models, we will emphasize them in this post. In the next section, we will discuss the key components of end-to-end speech-to-speech models.

2.1- Key components of end-to-end speech-to speech models

There are three main components within an end-to-end speech-to-speech model, namely the speech encoder, the LLM (decoder-only transformer), and the vocoder, as illustrated in Figure 1.

Specifically, the speech encoder processes raw audio waveforms into a set of audio embeddings. In multimodal systems, additional modalities, primarily text, are often required in conjunction with audio. A crucial concept in such systems is the alignment of text and speech modalities to ensure coherent integration. While we will not delve deeply into alignment techniques in this post, we include both text and audio modalities in our illustrations to provide a realistic example of a speech-to-speech model.

One possible way of aligning text and audio modalities is to concatenate the audio embeddings with the text embeddings, achieved by expanding the embeddings matrix of the model to include a new set of audio token embeddings. A mixed sequence of text and audio embeddings is then fed as input to the language model, which performs next-token prediction in an autoregressive manner, generating either text or audio tokens as needed. This approach is used, for instance, in AudioPaLM.

Finally, the vocoder converts the audio tokens produced by the language model back into high-quality audio waveforms, enabling natural and seamless speech synthesis.

2.1.1- Audio encoders

The audio encoder is a critical component in end-to-end speech-to-speech models, with several types of encoders employed depending on how the speech is represented. These encoders can be designed for two types of representations: discrete representation and continuous representation, as illustrated in Figure 2.

Discrete representation

In the case of discrete representation, the audio encoder is also referred to as an audio tokenizer. It processes continuous audio signals (waveforms) by first encoding them into latent representations and then converting these latent representations into discrete tokens. These tokens represent the audio in a compressed, quantized form and can be directly fed into the LLM for further processing. Models like AudioPaLM and Moshi utilize this approach.

Current models primarily use three main audio tokenization methods:

• Acoustic tokens, which focus on capturing sound patterns and low-level acoustic details. These are typically generated by audio encoders like SoundStream, a neural audio codec designed for audio compression and synthesis.
• Semantic tokens, which emphasize the meaning and linguistic content of the speech. These are extracted by models like HuBERT (Hidden-Unit BERT), which learns high-level semantic representations from speech for tasks like speech understanding.
• Mixed tokens combine elements of acoustic and semantic tokens, offering a balanced representation that captures acoustic, semantic, and paralinguistic features. Models like AudioLM utilize both acoustic and semantic tokens in tandem to ensure meaningful and high-quality speech synthesis and translation.

To further understand the distinctions between audio tokenization methods, Table 1 summarizes the characteristics, objectives, and applications of acoustic, semantic, and mixed tokens for clearer differentiation.

Continuous representation

In the case of continuous representation, the audio encoder extracts speech features that are unquantized, real-valued representations of speech signals existing on a continuous scale. These features capture fine-grained, nuanced aspects of speech that might otherwise be lost during discretization. To make these features compatible with the LLM, a feature projection layer, i.e., an audio projector is required to map the extracted audio features into continuous input embedding vectors, which are then transmitted to the LLM for further processing. An example of a model using this approach is SALMONN.

2.1.2- Vocoder

The other key element in end-to-end speech-to-speech models is the vocoder, also referred to as the Token-to-Speech Synthesizer. The vocoder’s primary role is to convert discrete or continuous audio tokens generated by the LLM back into high-quality, natural-sounding audio waveforms. A typical vocoder used for this purpose is HiFi-GAN, known for its ability to generate high-fidelity waveforms from token sequences. The choice of vocoder significantly impacts the quality and expressiveness of the synthesized audio, making it an essential part of the speech synthesis pipeline.

Capabilities and practical considerations

Now that we understand the basics of speech-to-speech generative AI models, let’s discuss the ideal capabilities and key considerations you should think of when building your applications with these models.

📜N.B: While this section primarily focuses on capabilities specific to voice, it’s important to remember that other general capabilities required for LLMs — such as factuality, reasoning, and memory — remain equally critical for creating effective and reliable speech-to-speech systems.

1. Low latency

Latency is one of the most critical factors in speech-to-speech systems, directly affecting user experience. Achieving low latency ensures that interactions feel natural and fluid, maintaining a good perceived responsiveness. While there are numerous latency metrics, the most important ones for speech systems include:

• Time to First Token (TTFT): Measures how quickly the system begins generating output after receiving input. A lower TTFT ensures faster responsiveness, which is crucial for maintaining conversational flow.
• Interruption Time: Refers to how swiftly the system can process and respond to user interruptions during ongoing speech. This is essential for natural turn-taking in dialogue.
• Average Latency: Represents the total processing time from input to output. Consistently low average latency enhances the overall experience, especially in real-time applications.

2. High-quality voice output

Beyond low latency, high-quality voice ensures user engagement, clarity, and satisfaction. From a quality perspective, ideal capabilities include:

2.1. Naturalness and speech intelligence

A good-quality speech-to-speech system should have naturalness and speech intelligence capabilities, such as emotion recognition, multilingual support, accent recognition, or sarcasm detection. These capabilities can be achieved through advanced voice encoders and vocoders, which capture subtle speech features such as intonation, rhythm, and prosody, as well as emotions like joy, sadness, or excitement.

2.2. Voice adaptability with prompt engineering

Contextual prompting for voice is an essential capability that enables easy adaptation of voice output features — such as tone, emotion, and style — using natural language prompts, without the need to explicitly define SSML (Speech Synthesis Markup Language) tags. For example, the prompt can adapt the voice to convey empathy in mental health applications, excitement in promotional scenarios, or professionalism in corporate interactions. For instance, generating a “Happy New Year” message with a dynamic voice could be as simple as this:

tone = "cheerful"
emotion = "joyful"

prompt = f"Generate a Happy New Year message with a {tone} tone and a {emotion} emotion. 

2.3. Speaker flexibility

Speaker flexibility is another essential component of high-quality voice output. By offering a variety of speakers — differing in gender, accent, or age — applications can cater to diverse user preferences and contexts. This flexibility is particularly important in areas such as customer service, where certain voice characteristics can improve user comfort, or in multilingual systems, where accents aid comprehension. Speech-to-speech systems that provide a range of speaker options ensures adaptability and inclusivity of diverse audiences.

3- Barge-in and full-duplex interaction

Barge-in refers to a user’s ability to interrupt a system’s speech output with their own speech input. This capability is essential for achieving full-duplex communication, where both the user and the system can simultaneously listen and speak, closely mimicking natural human conversational dynamics. Unlike text-based models, which typically operate in a half-duplex manner, full-duplex interaction is particularly important in voice-based systems to enable seamless, real-time interactions.

Recent advancements in speech-based models have aimed to incorporate full-duplex capabilities through techniques such as Voice Activity Detection (VAD), Parallel Stream Processing or Interleaved Token Modeling

4- Streaming

Streaming means the system should not wait for a lengthy audio segment to be processed before generating a response. Instead, the model typically operates on a chunk-based mechanism, dynamically processing and generating audio in real time, one chunk at a time.

Streaming functionality is closely tied to the need for low latency, and both are essential for seamless real-time interactions and overall system responsiveness.

5- Agentic capabilities

Agentic capabilities empower speech-to-speech models to autonomously perform tasks, make decisions, and interact with external tools or environments based on user input. While these functionalities are valuable for LLMs in general, they are particularly crucial for speech-to-speech models, enabling them to access external knowledge and be utilized in advanced applications beyond simple dialogue. Examples of agentic capabilities include accessing external tools, APIs or Databases through function calling or executing code.

6- Multimodal capabilities

Multimodal capabilities enable speech-to-speech systems to process and integrate inputs from multiple modalities, such as text, images, and video, alongside speech. This enhances context understanding, broadens use cases in areas like education and healthcare, and increases accessibility for users with disabilities. For instance, a system might analyze a photo and verbal description simultaneously to offer precise solutions. Despite challenges like data alignment and computational demands, these capabilities are driving the evolution of more intuitive systems, making interactions richer and more human-like.

Evaluation

As systems grow in complexity, evaluating speech-to-speech systems is a challenging task, as it must account for linguistic, emotional nuance, real-time performance and interactional factors. Effective evaluation requires a combination of approaches to capture both technical precision and user experience. Evaluating speech-to-speech systems remains a less advanced area of research compared to system development, with ongoing efforts to establish comprehensive and reliable methodologies. Available evaluation methods are typically divided into human-based assessments and automatic approaches, which include benchmarks and metrics.

📜N.B: This section aims to provide examples and is not an exhaustive list. Similarly to the capabilities section, we focus on metrics specific to voice. Common metrics used for LLMs, including LLMs as a judge, are not covered in this post.

1- Human evaluation

Human evaluation is critical for assessing subjective qualities like naturalness, expressiveness, and overall user experience. The Mean Opinion Score (MOS) is an example of metrics used, where raters score audio samples on a predefined scale based on perceived quality, focusing on intelligibility, emotional expressiveness, and fluency. However, while effective in capturing human-centric insights, MOS is resource-intensive and influenced by rater bias.

2- Automatic evaluation

Automatic evaluation provides objective measures to complement human assessment. It can be divided into two main components:

2.1. Benchmarks

Benchmarks such as VoiceBench, SUPERB (Speech Processing Universal PERformance Benchmark), AudioBench or AIR-Bench standardize the evaluation of speech-to-speech systems by providing pre-defined datasets and tasks. These benchmarks enable fair comparisons across models and ensure consistency in assessing performance across areas like speech recognition, translation, and synthesis.

2.2. Metrics

Metrics quantify specific aspects of system performance. For semantic accuracy, metrics like SpeechBERTScore, SpeechBLEU or SpeechTokenDistance can be used.

Acoustic quality metrics include Mel Cepstral Distortion (MCD) which evaluates the difference in spectral features between generated and reference speech or Log F0 Root Mean Square Error (RMSE) which measures prosody accuracy by comparing fundamental frequency (F0) patterns. Real-time performance is evaluated through latency metrics, while interaction metrics such as Turn-taking accuracy assess responsiveness and coherence in dynamic conversational scenarios. Figure 3 provides examples of evaluation metrics.

Use cases and best practices

Speech-to-speech generative AI models have transformative applications across various domains. Examples of use cases include conversational assistants and agents, such as virtual customer service representatives or personal assistants. They are also valuable for language learning and testing, allowing users to practice pronunciation and comprehension in different languages. Effective integration with other systems, such as multimodal frameworks or text-based LLMs, extends functionality to diverse domains: education, by enabling interactive language tutors or immersive learning environments; healthcare, by supporting voice-enabled patient assistance and telehealth services; entertainment, by enhancing gaming experiences and virtual reality interactions; and sports, by providing real-time commentary or personalized coaching through voice interfaces.

Developing speech-to-speech applications is a complex task that requires careful planning and execution. Here are some practical recommendations to guide you:

1- It’s not just the model, think of the full stack — From the user interface, data storage, APIs, to infrastructure, every layer impacts performance. Adopt a long-term vision and choose a stack that offers you flexibility to integrate with existing and new technologies, scale with demand, and adapt to changing requirements and future advancements.

2- Thoroughly test, evaluate and monitor your system — Validate every component of your voice system across multiple aspects: accuracy for accents, languages, clarity, naturalness, hallucinations and behavior in noisy conditions. Build scalable test protocols for real-world scenarios to measure latency and the ability to handle interruptions. Perform scalability tests for stress and edge cases. Include user feedback, accessibility checks. Regularly update tests to cover new features and keep monitoring over time.

3- Responsible AI is even more critical for speech systems — It’s not just about latency or voice quality; it’s also about safety and trust. Implement strong guardrails to prevent harmful or biased outputs, ensure user privacy, and comply with regulations. Speech systems are accessible and interact directly with users — making ethical considerations non-negotiable.

4- Trade-offs are key: optimize for your use case — Noticed the diverse capabilities of generative AI speech models? You don’t need all of them to be perfect. Consider trade-offs to prioritize what matters most for your application. For example, you might accept slower latency for higher accuracy or sacrifice flexibility for better real-time performance in some use cases.

5- You need the right mindset and expertise — Speech-to-speech systems lies at the intersection of multiple disciplines including multimodal AI, software engineering, and user experience design. Fostering collaboration is critical, as interdisciplinary problem-solving is essential to creating an effective user experience and adopting the technology effectively. Generative AI is also a fast-moving field, requiring a growth and learning mindset to stay ahead of advancements. Invest in the right talent and build teams that excel in cross-functional expertise, embrace diverse perspectives, and communicate effectively.

Conclusion

The rise of speech-to-speech generative AI is certainly opening up a range of new possibilities and can assist us in countless ways. However, this potential can only be fully realized if these technologies are built responsibly, users are educated on ethical use, and awareness is raised about the broader ethical implications and societal impacts. This field is constantly evolving, and what we’ve covered here is not exhaustive — it’s just the starting point. Staying informed about the latest innovations and engaging with the research community are essential for developing a comprehensive understanding.

How to build your speech-to-speech applications with Google Cloud?

1. Explore the multimodal live API
   Discover how the Multimodal Live API, powered by Gemini, enables low-latency, bidirectional voice and video interactions. This API is designed for seamless, real-time communication, making it ideal for applications like virtual assistants, interactive education, and immersive entertainment.
2. Check out Gemini 2.0 advanced native audio capabilities
   Watch the video below to see Gemini 2.0 in action.

📜About the Author

Dr. Wafae Bakkali is a Staff Generative AI Specialist, Blackbelt at Google. With over 12 years of experience, she has a proven track record of combining deep technical expertise and leadership with strategic business insights to drive impactful outcomes. Through her current role at Google and former roles at reputable organizations such as AWS, Dr. Bakkali has successfully guided organizations across various industries in adopting and building AI solutions.

She holds a PhD from IMT Atlantique, France, and served as a postdoctoral researcher at Centrale Supélec, Paris-Saclay University, France.

Dr. Bakkali is also an active contributor to the AI community. She regularly shares her expertise as a speaker at international tech conferences and mentors participants in AI hackathons, tech, and startup events.

🔗Connect with me on LinkedIn: https://www.linkedin.com/in/wafae-bakkali/

Speech-to-Speech Generative AI: From Theory to Practice was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.