Interest in Microsoft Fabric has grown quickly among organisations with complex data estates. The proposition is a single platform for data engineering, warehousing, data science, real-time intelligence and reporting, built on one storage layer and one model of governance. For organisations that have accumulated separate tools, teams and copies of the same data over many years, the prospect of consolidation is compelling.

A demonstration establishes that the platform can do what it promises under controlled conditions. The decision to extend it across an organisation requires evidence of a different kind — what the platform will cost at production volumes, whether the organisation’s data is ready to move, whether governance will hold as more teams arrive, and whether people will change the way they work.

This matters because the ability to evidence value remains a persistent weakness in data investment. In Gartner’s most recent survey of chief data and analytics officers, 30% named the inability to measure the impact of data, analytics and AI on business outcomes as their top challenge, and only 22% of organisations had defined, tracked and communicated business impact metrics for most of their use cases. A platform decision of Fabric’s scale deserves a stronger foundation. In our work with organisations evaluating Fabric, we have found that the most reliable foundation is a pilot designed from the outset as a decision instrument, tested against a meaningful business problem, measured from the first day, and concluded with an explicit choice to scale, refine or stop.

Our earlier article explored what Fabric consolidates and why that consolidation matters. This article considers how organisations can establish whether it will deliver value for them.

Starting with the business problem

The credibility of a pilot depends largely on the problem it is asked to solve. McKinsey has described how organisations often fall into what it calls “pilot purgatory”, with teams launching proofs of concept that have no chance of scaling, and stakeholders pursuing varied use cases that require an entire architecture to be built before any value can be realised. A Fabric pilot built around a technical showcase, such as migrating a sample dataset, risks the same outcome. It produces results that are interesting to technical teams and difficult to translate into a case a finance director can weigh.

The strongest pilots begin with an operational problem the business already recognises. In our experience, good candidates share a number of characteristics. They have a named business owner who feels the problem today and will judge the result. They carry a cost or value that can be expressed in the organisation’s own terms, whether hours of manual effort, delayed decisions, regulatory exposure or lost revenue. They involve data at representative volume and complexity, drawn from the systems the organisation runs. They also cross team boundaries, because much of Fabric’s benefit lies in bringing engineers, analysts and business users into a shared environment, and a pilot confined to a single team cannot test that effectively.

Scope and duration deserve equal consideration. A pilot needs to run long enough to produce a representative picture of how the platform will be used and consumed, and short enough to reach a decision while the business question remains live. For most organisations, a period of two to four months strikes that balance.

Month-end financial reporting offers a useful illustration to move from. In many large organisations, the close still depends on extracts from several source systems, manual reconciliation in spreadsheets and a small team working long hours at the start of each month. The problem is visible to senior leadership, its cost can be measured in time and error rates, and solving it requires data engineering, modelling and reporting to work together. It is the kind of problem that reveals whether a platform can deliver in a live environment.

Six measures of a credible pilot

A pilot becomes a decision instrument when it is judged against measures agreed before any work begins. We recommend six, each with a baseline taken at the outset and a threshold that would justify scaling. Agreeing both in advance gives the business owner, the technology team and the executive sponsor a shared definition of success, and it protects the eventual decision from being shaped by enthusiasm or by hindsight.

Business outcomes: They should be expressed in the language of the business owner, such as the number of days taken to close the month or the time between a question being asked and a decision being made. Technical measures such as query performance matter to the extent that they move these outcomes.

Data quality: This establishes whether consolidation improves the reliability of the information people use. This measure has gained weight as organisations look to build AI capabilities on their data platforms. Gartner predicts that through 2026, organisations will abandon 60% of AI projects that are not supported by AI-ready data, and McKinsey reports that eight in ten companies cite data limitations as a barrier to scaling agentic AI. A pilot that demonstrates measurable improvement in data quality strengthens the business case well beyond reporting.

Governance: This considers whether the platform can be controlled as effectively as it can be used. The pilot should test access control, lineage, sensitivity labelling and data loss prevention against the organisation’s own policies and regulatory obligations. The more important question concerns what happens as adoption grows, since controls that work well for a single team can weaken once many teams and workspaces are involved.

Adoption: This reflects whether people change the way they work. Indicators include the number of active users, the reuse of shared datasets and models, and the retirement of spreadsheets and duplicate reports that the platform was intended to replace. Reuse deserves particular attention. McKinsey’s research on data products found that reusable data assets can accelerate the capture of a use case’s value by as much as 90 percent, which makes early evidence of reuse one of the strongest predictors of value at scale.

Capacity: This describes how the pilot consumes the platform’s shared compute resources. All Fabric workloads draw on a common pool of capacity, so the pattern of consumption observed during the pilot is the most reliable guide to sizing a production environment. Microsoft provides dedicated tooling for monitoring utilisation and identifying the operations that use the most compute, and we advise putting that monitoring in place on the first day, so the evidence accumulates across the full pilot.

Cost: This draws the other measures together into a projected cost of running the platform at scale. It should account for capacity, storage growth, licensing and the effort required to operate and support the platform. The timing of commercial commitments is an important consideration here. Longer-term capacity commitments can reduce unit costs, and they are best made once the pilot has produced a dependable picture of consumption.

For the month-end reporting example, a pilot scorecard might take the following form:

A pilot should conclude with one of three decisions, and each represents a valid outcome.

Scale:

A decision to scale is appropriate when the pilot has met its thresholds for business outcomes, data quality and cost, and when governance and adoption have held up under realistic conditions. It should be accompanied by a clear view of which use cases follow, the capacity they will require and the operating model needed to support them.

Refine:

A decision to refine is appropriate when the pilot has demonstrated real value while revealing weaknesses that would grow at scale. Strong business outcomes accompanied by limited adoption are a common example, as is a promising result offset by a consumption profile that would be costly to sustain. Refinement means addressing those weaknesses in a focused second phase before committing further, whether through optimisation, a stronger governance model or investment in skills.

Stop:

A decision to stop is appropriate when the evidence shows that the platform will not deliver sufficient value for the organisation at this time. Existing platforms may already meet the need, the data estate may require foundational work before any platform can succeed, or the projected cost may exceed the benefit. We regard a well-evidenced decision to stop as a successful pilot. It protects the organisation from a far larger commitment and leaves a clear record of the conditions under which the question could be revisited.

Preparing for scale

A successful pilot establishes the business case. Preparing the organisation to realise that case at scale requires further groundwork, and in our experience the organisations that scale most smoothly address four areas before the rollout widens.

The first is capacity planning, informed directly by the consumption evidence gathered during the pilot and structured to separate development and production workloads where this is justified. The second is the design of workspaces and data domains, which should reflect how the organisation is structured, so that ownership of data is clear and responsibility for governance can be delegated with confidence. The third is a formal governance model covering how workspaces are created, how datasets are certified and how sensitive information is protected. The fourth is a skills and adoption plan that addresses the full range of users, from engineers adopting new tooling to business teams moving from spreadsheets to shared models.

Sequencing is a further important consideration. Organisations that select their next wave of use cases using the same six measures, prioritising business value and data readiness, tend to build credibility with each phase. Those that attempt to migrate their entire estate at once often find that effort is spread too thinly for any single area to demonstrate clear value.

Evidence before commitment

Microsoft Fabric has the potential to simplify data estates that have become complex over many years, and to bring engineering, analytics and business teams into a shared way of working. Whether that potential is realised depends on each organisation’s data, governance, people and budget, as much as on the capabilities of the platform itself.

Audacia’s approach to Fabric evaluation follows the sequence we apply across our consultancy work. We begin by understanding the business problem, the current data estate and the outcomes that matter to the organisation. We then define the pilot, its scope and its measures with the business owner, recommending a decision based on the evidence the pilot produces, whether that means scaling, refining the approach or pausing until the conditions are right. The organisations that gain most from Fabric are those that treat the first pilot as a test of the business case as well as the technology, and that are prepared to act on whatever the evidence shows.

Originally published at https://audacia.co.uk.

Microsoft Fabric: How to Prove the Business Case Before You Scale was originally published in CodeX on Medium, where people are continuing the conversation by highlighting and responding to this story.

Green tests don’t tell an AI agent what it must leave alone. A one-page preservation contract does.

Here is a failure many teams will meet this year. An agent fixes a bug in a records service. On the way, it “simplifies” a shared helper and removes a filter that looks redundant. Every test passes.

A week later, the CSV export shows users rows they were never meant to see. The export called the same helper, and no test checked whose rows ended up in the file.

The agent did exactly what it was asked. Nobody had written down what it must not change. That gap, not the agent, is what this article is about.

Every change has two intents

Every change to existing code carries two intents: what should change, and everything that must stay the same. Tickets capture the first. The second stays implicit, because to the people who know the system it feels obvious.

It is not obvious to an agent. The agent doesn’t know which teams call the helper, which incident created that odd-looking filter, or which customer depends on the sort order. It improves the code in front of it.

Tests don’t close the gap on their own. They catch only what someone once thought to check, and in old systems much of the behavior people rely on is checked by nobody.

Bug-fix specs give a clean way to state both intents. When the bug condition C holds, the fix must make the expected result P true. When C does not hold, the code must behave exactly as before:

The second half is the one we skip. The rest of this article is about writing it down.

Hidden contracts are everywhere

With enough users, every observable behavior of a system becomes something somebody depends on, whatever the documentation promises. Engineers know this as Hyrum’s Law.

In practice, hidden contracts look boring:

• the order of results
• an empty list versus null (or None)
• rounding rules
• date and time formats
• error and status codes
• which rows a user is allowed to see

The cheapest way to find them is to look at who calls the code you are about to change. One search for a shared helper might turn up this:

grep -rl "fetch_records(" src/

Caller

Relies on

Covered by a test

src/export/csv.py

only rows the user may see

No

src/mobile/feed.py

newest first

No

src/billing/run.py

an empty list, never None

Yes

src/api/partner.py

ISO 8601 dates

No

src/reports/tax.py

amounts rounded half-up

Yes

src/web/status.py

404 for a missing record, never 500

No

One function, six hidden contracts, four with no test. The example is illustrative, but the shape is common.

Treat an empty search result as limited knowledge, not proof. Reflection, configuration, scheduled jobs and external API clients rarely show up in a grep.

The preservation contract

A preservation contract is one page, written before the agent starts, with six fields. Each field closes a different shortcut an agent can take to “pass” a task.

Field

What it says

The shortcut it blocks

Condition

When the change applies

Quietly widening the problem

Expected result

What must be true afterwards

Hiding the error instead of fixing it

Protected behaviors

Guarantees this change must not break

“Cleaning up” a neighbor

Allowed scope

Files and modules it may touch

Silent changes elsewhere; anything outside is reported

Fix proof

A test that fails on the old code and passes after

Claiming a fix nobody can see

No-regression proof

Tests that pass both before and after

Breaking what nobody checked

For the helper above, a filled-in contract fits on one screen:

# change-contract.md: duplicate rows in fetch_records

Condition:        merged accounts return each record twice
Expected result:  every record is returned once
                  (not "the error is gone")
Protected:
  - export returns only rows the user may see   owner: Data
  - feed order stays newest first                owner: Mobile
  - empty result is [], never None               owner: Billing
Allowed scope:    records/ only. Anything else is reported, not changed.
Fix proof:        test_no_duplicate_rows   fails on base, passes after
No-regression:    6 contract tests         pass on base, pass after

The contract doesn’t replace tests, CI or review. It gives the reviewer something concrete to hold the diff against.

Test the old code first

The order of work matters more than the template: protective tests are written and run against the old code, before anyone touches the fix.

Tests written after the fix describe the new code. When the agent writes both the change and its tests, a green suite only proves that the code agrees with itself.

1. Check out the base commit.
2. Write the fix test and confirm it fails.
3. Write the protective tests and confirm they pass.
4. Commit the tests on their own, before any change to the code.
5. Let the agent make the change, then run everything.

Michael Feathers calls tests like these characterization tests: they record what the code does today, not what it should do. Keep them out of the agent’s reach while it works, so it can’t “fix” a test instead of the code.

After the change, the results should read like this:

Test

Old code

After the agent’s fix

no_duplicate_rows (fix proof)

Fail

Pass

export_visible_rows_only

Pass

Fail

feed_newest_first

Pass

Pass

billing_empty_list_not_none

Pass

Pass

Two outcomes are good: the fix test turns green, and every protective test stays green. A protective test that changes color means the change left its scope, so narrow it and run again.

In this run, the export test caught the same removed filter that slipped through in the opening story.

Commitment or accident?

A characterization test records a bug as faithfully as a feature. So every recorded behavior needs one more decision: is it a commitment someone relies on, or an accident that is free to change?

Excel shows how a bug becomes a commitment. It still treats 1900 as a leap year because Lotus 1–2–3 did, and Microsoft explains that a fix would shift almost every existing date by one day and break formulas.

G. K. Chesterton’s fence makes the same point: before you remove a fence, find out why it was put up. The contract is where that answer gets written down, before the agent removes the fence.

A named person makes each call, not the agent and not “the team”:

Recorded behavior

Decision

Who decides

Next step

Export shows only visible rows

Commitment

Data team lead

Protective test

Feed is sorted newest first

Not decided yet

Mobile product owner

Keep the current order until decided

Empty result is [], never None

Accident, after checking every caller

API owner

Test the new behavior and note it in the spec

The agent gets no permission to change a behavior that hasn’t been through this table.

Grow the spec where the code changes

You don’t need to document the whole system before the first change. Each contract adds a small delta to a living spec, and the spec grows exactly where the system moves.

A delta records three things: what this change added, what it modified, and what it removed. It also records the guarantees this change wrote down for the first time, such as “the export returns only rows the user may see.”

That second part matters. On day one a legacy spec is empty, so nothing is protected by it yet. Each approved guarantee moves from the contract into the spec, tied to a test that keeps guarding it.

After a few months, the spec is dense around the hotspots and thin where nothing changes. A full inventory still makes sense when it is the deliverable itself, such as a platform migration or an audit.

Start with three changes

Try the contract on three changes of growing difficulty, then decide whether to keep it. Each one teaches something different:

1. A small bug fix. The team learns the order: boundary first, protective tests on the old code, then the fix.
2. A feature next to a busy file. The contract lists real callers and past incidents, or says plainly that there is no data.
3. A refactoring. The commitment-or-accident table gets filled in for real, with a name on every row.

Three signals say it is worth spreading: a protective test changed color and narrowed a patch, reviewers quote contract fields instead of impressions, and deltas reach the spec without extra work.

Three signals say stop or simplify: contracts appear for typo fixes, protected behaviors are copied from a template instead of found, or the contract repeats what review already catches. Patch size is not the test, though. A one-line change to authorization or payments may still need the full page.

The takeaway

In legacy code, specify the change, not the system. Before the first prompt, write down what must not change, prove it on the old code, and let a named person decide what counts as a commitment.

Your next ticket in old code can get a change-contract.md before the agent writes a single line. One page is enough to make a green build mean something.

Which behavior in your system turned out to be a commitment only after someone changed it?

This article builds on Jaroslaw Wasowski’s piece “Spec-Driven Development — Brownfield Spec and Preservation Contract” ([link to the original article]), which develops the preservation contract in more depth.

Sources

• Hyrum’s Law, explained by Unified Streaming
• Microsoft: Excel incorrectly assumes that the year 1900 is a leap year

Preservation Contracts: Safer AI Agent Changes in Legacy Code was originally published in CodeX on Medium, where people are continuing the conversation by highlighting and responding to this story.

For example:

• read temperature, humidity, or pressure
• understand the current physical context
• answer the user
• optionally change the robot expression to happy, curious, or surprised

The important architectural constraint was that Claude should not access sensor drivers, display code, or robot internals directly. That is where MCP becomes useful.

This diagram shows what i wanted to obtain:

Phos is the name of my simple robot developed using Python on Raspberry Pi.

MCP in one minute

The Model Context Protocol provides a standard way for an AI client to discover and invoke tools exposed by an external system. For this use case, only two transport models matter.

stdio

A local MCP client starts a process and communicates with it through:

stdin / stdout

This model is convenient for desktop tools because the MCP server looks like a local executable.

Streamable HTTP

The MCP server runs independently and exposes an HTTP endpoint. Conceptually:

AI client
    ↓
MCP over HTTP
    ↓
remote service

This model fits a Raspberry Pi much better. The Pi is a continuously running physical device. Its lifecycle should not depend on whether Claude Desktop is connected. That leads to the architecture used here:

The Raspberry Pi exposes MCP over HTTP. Claude Desktop gets the stdio interface it expects. The bridge connects the two.

Why use a bridge?

A first implementation could make Claude Desktop connect to a process running directly on the Raspberry Pi over stdio. That works, but it creates an undesirable lifecycle dependency:

Claude starts MCP
      ↓
MCP starts robot runtime

For a physical device, I prefer the opposite model:

Raspberry Pi starts
      ↓
robot runtime starts
      ↓
MCP endpoint becomes available
      ↓
Claude may connect later

The robot continues operating when Claude disconnects. The local bridge solves the compatibility problem without changing that architecture. Its only responsibility is:

MCP stdio
→
MCP Streamable HTTP

It should not contain robot logic. It should not know how sensors work. It should not implement copies of Raspberry Pi functions. It simply forwards MCP operations between Claude Desktop and the remote MCP server.

A practical setup looks like this:

The bridge can run on macOS, Linux, or Windows while the Raspberry Pi remains dedicated to the physical runtime.

The Raspberry Pi should expose semantics, not hardware

This is the most important part of the design. An MCP tool should not look like this:

read_bme280_register
set_gpio_pin
draw_pixel
write_led_buffer

Those are implementation details. Claude should see operations closer to the meaning of the physical system. For example:

get_environment
get_robot_status
get_physical_context
set_expression
speak
sleep
wake

The interaction path becomes:

Claude Desktop
      ↓
MCP
      ↓
semantic application layer
      ↓
runtime services
      ↓
hardware

Not:

Claude Desktop
      ↓
MCP
      ↓
sensor driver

This keeps the agent independent from the hardware implementation. A sensor can change later without changing the MCP contract. The same applies to the display. Claude does not need to understand how the eyes are rendered. It only needs a semantic operation such as:

set_expression("curious")

The Raspberry Pi decides how that expression is actually displayed.

Building a semantic physical context

A useful MCP tool can expose a compact description of what the Raspberry Pi currently perceives. For example:

{
  "environment": {
    "temperature_c": 22.8,
    "humidity_percent": 48,
    "pressure_hpa": 1015
  },
  "air_quality": {
    "available": true,
    "condition": "good"
  },
  "presence": {
    "detected": true
  },
  "robot_state": "idle"
}

The exact structure depends on the application. The important property is that this model should describe:

what the physical system currently perceives

rather than:

which driver produced each value

This layer can also include interpreted values. For example:

{
  "temperature_c": 28.4,
  "temperature_condition": "warm"
}

That allows the agent to reason over a stable semantic model instead of repeatedly interpreting low-level device data.

Example: “How is the room?”

Consider a simple interaction. The user asks Claude Desktop:

How is the room?

Claude does not need direct access to the Raspberry Pi. Instead, it calls an MCP tool such as:

get_physical_context

The request travels through:

Claude Desktop
      ↓
stdio
      ↓
MCP bridge
      ↓
Streamable HTTP
      ↓
Raspberry Pi MCP server
      ↓
semantic application service
      ↓
sensor services

The Raspberry Pi might return:

{
  "environment": {
    "temperature_c": 27.1,
    "humidity_percent": 61,
    "pressure_hpa": 1008,
    "condition": "warm"
  },
  "air_quality": {
    "condition": "good"
  }
}

Claude can now produce a natural-language response such as:

The room is fairly warm at 27.1°C, humidity is moderate, and the air quality looks good.

Then Claude can decide that a visual reaction is appropriate. For example:

set_expression("surprised")

or:

set_expression("happy")

That request follows the same MCP path back to the Raspberry Pi. The display changes expression, while the robot runtime remains responsible for the actual rendering. The complete interaction is:

User
  ↓
Claude Desktop
  ↓
get_physical_context
  ↓
Raspberry Pi sensors
  ↓
semantic context
  ↓
Claude reasoning
  ↓
natural-language response
  ↓
set_expression
  ↓
Raspberry Pi display

This is the pattern I wanted: the AI reasons about the physical world without becoming the hardware controller.

Tool discovery should stay dynamic

If the bridge hardcodes every Raspberry Pi tool, it becomes tightly coupled to the robot implementation. A better model is:

The same applies to tool calls.

This means adding a new MCP capability on the Raspberry Pi does not necessarily require changing the bridge.

The Pi remains the authoritative source of the available MCP interface.

Lifecycle ownership matters

The architecture I ended up using follows one rule:

The Raspberry Pi owns the physical runtime.
MCP only exposes it.

In one real implementation, PHOS, the process model is effectively:

1 Raspberry Pi process
1 robot runtime
1 application service graph
1 semantic MCP surface
N MCP client connections

An MCP connection never creates another robot runtime. Claude can connect or disconnect without changing the lifecycle of the physical system. That separation becomes increasingly important as more clients are added.

Why this architecture works well

The main benefit is not simply that Claude Desktop can invoke tools on a Raspberry Pi. The benefit is the boundary. Claude gets a semantic description of the physical system:

temperature
pressure
air quality
presence
state
available actions

The Raspberry Pi keeps control over:

drivers
hardware
runtime
state
rendering
lifecycle

The result is a clean separation:

Claude
= reasoning

Raspberry Pi
= embodiment

MCP
= semantic interface

Using MCP to Control a Raspberry Pi from Claude Desktop was originally published in CodeX on Medium, where people are continuing the conversation by highlighting and responding to this story.

Scraping scripts rarely fail on day one.

They fail quietly on day ten.

The script runs. No exception. A CSV appears. And somewhere in it, 24 rows are missing. Nobody notices until the data is already in a dashboard.

If you’re:

• collecting product, pricing or catalogue data,
• using Claude Code to write your scrapers,
• or tired of scripts that “work” but return half the data,

This matters.

I gave Claude Code a plain-language scraping goal and Gaffa’s AI skills, then let it go from zero to a working web scraping workflow: discovery, a paginated scraper, CSV export and verification. Then I tried to break it.

Every number, error and output below comes from real runs.

The real problem with AI-written scrapers

Writing a scraper is the easy part. Claude can do it in one shot.

The hard part is everything around it:

• Pagination that stops one page early
• Pages that load slowly some of the time
• Errors that only happen once in a while

None of these throw an error. The script finishes, the CSV looks fine, and the data is wrong.

You don’t find these bugs by reading code. You find them by running the scraper against the real site and checking the output against something you trust.

A scraper needs proof, not just code

The fix isn’t a smarter prompt.

It’s a workflow where the agent can discover the site, build against the real pages, and debug failures from real request data. With checks that tell you when the output is wrong.

Gaffa’s AI skills cover exactly those three steps:

• /gaffa-find: give it a URL and a goal, and it explores the site with a credit budget until it finds the answer
• gaffa-authoring: loads automatically whenever you ask for scraping code, and makes Claude capture the real page before writing a selector
• /gaffa-debug: pass it a failing request id (brq_...) and it diagnoses what went wrong

Under the hood, Gaffa is a browser automation REST API: you send a URL and a list of actions (click, wait, capture_element, loop...) and it runs them in a real browser.

👉 You can create an account at gaffa.dev and follow along.

Setup in one command

If you read my previous article on connecting Claude Code to Gaffa, you already have this. If not, it’s one command in your project folder:

npx @gaffa-dev/cli install --tools=claude-code --scope=project -y

Installed the gaffa skills (version 0.9.0) into 1 place:
  ./.claude\skills  (project)  gaffa-authoring, gaffa-debug, gaffa-find, gaffa-support

Registered the gaffa-docs MCP server:
  Claude Code  ./.mcp.json  added

Then set GAFFA_API_KEY as an environment variable. The skills never read the key from anywhere else.

The target for this test is books.toscrape.com, a sandbox catalogue built for scraping practice: 1,000 books, 50 categories, classic “next” pagination.

Phase 1: Discover the site with /gaffa-find

I didn’t give Claude a URL for the category, a selector or a page count. Just the homepage and a goal:

/gaffa-find https://books.toscrape.com/ — Goal: I want to collect every book
in the 'Mystery' category. Find the category page URL, how many books it
contains, how many listing pages it spans and how pagination works, and which
fields each book card exposes (title, price, rating, availability, link).

Rendered from the real session transcript.

The skill runs a loop: recon, hypothesis, targeted extraction, validation. With hard caps: 15 requests and 50 credits per run, 5 iterations maximum.

For recon, it sent a browser request that turns the homepage into Markdown. 3 credits, 5.8 seconds, and the answer was in the sidebar:

- [Mystery](catalogue/category/books/mystery_3/index.html)

From there it captured the category page twice: once as Markdown (to read “32 results” and “Page 1 of 2”) and once as raw HTML (to see the markup the Markdown strips out, like the star rating). Then it checked page 2 to confirm where pagination ends.

It also wrote a reasoning log to disk, .gaffa-find-<timestamp>.log, one entry per iteration: hypothesis, request id, credits, decision. You can audit exactly how it reached its answer.

The final result:

Rendered from the real session transcript.

Terminal state: success. 32 books, 2 pages, pagination through a relative li.next a link that disappears on the last page, and a selector for each field. One detail I wouldn't have caught: the visible title text is truncated with "…", so the full title has to come from the link's title attribute.

Cost of discovery: 9 credits.

Phase 2: Turn the findings into a workflow

Same Claude session, so it kept everything it had learned. This time I asked for a real workflow:

Rewrite it as books_to_csv.py: one single Gaffa browser request that
paginates with the loop action. Price as a number, rating as an integer
1-5, absolute product URL. Write a CSV. Add --category-url. Run it on
Mystery, verify 32 rows with no duplicates, then run it on a category
with 3+ pages and verify the count matches the 'N results' text.

Claude pulled the loop reference through the Gaffa docs MCP server (approved this time), then built the request.

One request, every page

This is the heart of the workflow. Instead of one request per page, a single request captures the results banner once and then loops: capture the product grid, click “next”, repeat until there is no “next” link.

"actions": [
    {"type": "capture_element", "selector": "form.form-horizontal", "custom_id": "results-banner"},
    {
        "type": "loop",
        "custom_id": "pagination-loop",
        "max_iterations": max_pages,
        "timeout": loop_timeout,
        "stop_on_fail": True,
        "continue_on_fail": True,  # the last page has no "next": that's how the loop ends
        "actions": [
            {"type": "wait", "time": 800, "custom_id": "settle"},
            {"type": "capture_element", "selector": "ol.row", "custom_id": "page-cards"},
            {"type": "click", "selector": "li.next a", "custom_id": "next-page", "timeout": 15000},
        ],
    },
],

Parsing happens in plain Python with regular expressions over the captured HTML. No LLM extraction. The skill steers Claude away from Gaffa’s parse_json action for structured pages like this one, because a deterministic parser is cheaper and returns the same output every run.

The check that caught a bug

The most valuable line in the script is not the scraper. It’s this:

status = "OK" if reported_total == len(books) else "MISMATCH"
print(f"page says {reported_total} results; scraped {len(books)} -> {status}")
if status == "MISMATCH":
    sys.exit(1)

The workflow compares its own output with the number the website reports. If they differ, it fails loudly.

Here is how phase 2 went:

Attempt 1: Mystery. 32 of 32 books, 32 unique URLs, banner matched. 3 credits.

Attempt 2: Nonfiction (110 books, 6 pages).

request: brq_VzZhqRu7EsTn47GhMyLqkSN76iJDA6
pages captured: 6
books written: 86 -> nonfiction.csv
page says 110 results; scraped 86 -> MISMATCH

All 6 pages were captured, but 24 books were missing. Claude broke it down page by page:

page 1: articles=20
page 2: articles=20
page 3: articles=20
page 4: articles=12
page 5: articles=14
page 6: articles=2

Pages 4 to 6 were captured while the browser was still loading them, right after the click. Claude added a short wait at the start of each iteration and re-ran: 110 of 110. Then re-ran Mystery to make sure nothing regressed.

Rendered from the real session transcript.

Without that count check, the script would have exited 0 and handed me a CSV missing 22% of the data.

Cost of phase 2: 6 billed requests, 18 credits.

Then I tried to break it

Claude said it was done. I didn’t take its word for it. I ran the script myself on categories it had never seen.

Travel (11 books) and Fiction (65 books, 4 pages) both matched their counts. Then I ran Nonfiction again:

pages captured: 2
books written: 40 -> nonfiction.csv
page says 110 results; scraped 40 -> MISMATCH

The loop stopped after page 2. The count check caught it again.

Failed and partial requests are part of scraping any live website. Pages get slow, layouts change, content loads late. What matters is how quickly you find out why. That’s the job of the third skill.

Phase 3: Diagnose with /gaffa-debug

I passed the request id to Claude:

/gaffa-debug brq_VzZj6m1Mqsb31bZEevh2ZHdUNVFxyh — it only got 40 of 110
books (MISMATCH). Diagnose why the loop stopped early, apply a minimal fix
that does not rely on luck, then verify on Nonfiction twice and Mystery once.

Rendered from the real session transcript.

The diagnosis was precise:

1. The request data showed the loop ran 2 iterations, and the second “next” click failed with action_timed_out.
2. Was page 2 really the last page? Claude fetched page-2.html directly (free, no Gaffa call). It has a "next" link to page 3. So the loop stopped early by mistake.
3. Why? On that run the target page took 27.9 seconds to load, against about 2 seconds on every other run. Live sites have slow moments, and the click only waited 5 seconds before giving up.

The patch was one value: raise the click’s timeout from 5,000 to 15,000 ms, the same value Gaffa's own pagination examples use. A real last page still ends the loop, just 15 seconds later.

Verification: Nonfiction 110 of 110 twice in a row with identical rows, Mystery 32 of 32. 13 credits.

The final result

I ran the final script one last time on my own:

218 books, 4 categories, 0 duplicates. Nonfiction, 6 pages in a single request, takes about 42 seconds end to end.

The CSV is clean, typed data ready for pandas or a spreadsheet:

title,price,rating,availability,url
Worlds Elsewhere: Journeys Around Shakespeare’s Globe,40.3,5,In stock,https://books.toscrape.com/catalogue/worlds-elsewhere-journeys-around-shakespeares-globe_972/index.html
The Five Love Languages: How to Express Heartfelt Commitment to Your Mate,31.05,3,In stock,https://books.toscrape.com/catalogue/the-five-love-languages-how-to-express-heartfelt-commitment-to-your-mate_969/index.html
Reasons to Stay Alive,26.41,2,In stock,https://books.toscrape.com/catalogue/reasons-to-stay-alive_959/index.html

What the whole thing cost

Claude Code ran on my Claude Pro subscription.

From here you can:

• schedule it with cron or Windows Task Scheduler, since it exits 1 on a mismatch
• loop over all 50 category URLs to build the full 1,000-book catalogue
• point the same pattern at any paginated listing: job boards, product grids, directories

What I’d do differently next time

Three lessons from this run, all of which apply outside Gaffa:

Put the verification in the script, not in your head. The “N results” check caught two silent failures that would otherwise have shipped. If the site tells you how many items it has, use it.

Debug from request data, not from guesses. /gaffa-debug didn't guess. It read how many iterations ran, which action failed and how long the page took to load, then checked the live page before changing a single value.

Fixed waits are guesses. The 800 ms wait happened to work, and the 5 s click timeout didn't when the site was slow. When you can, wait on a selector or give timeouts real margin.

Key takeaways

• /gaffa-find turned "collect every Mystery book" into URLs, selectors and pagination rules in 9 credits.
• One Gaffa request with loop scraped a 6-page category in about 42 seconds, and a built-in count check caught two silent data losses.
• /gaffa-debug traced a stopped loop to a 27.9-second page load and fixed it with a single timeout change.

Final thoughts

Claude Code writes scrapers fast. That was never the bottleneck.

The bottleneck was knowing whether the data is right.

Give the agent a real browser, skills that force it to look before it writes, and a workflow that checks its own output, and web scraping with Claude Code becomes something you can actually trust.

👉 Try it: create an account at gaffa.dev, run npx @gaffa-dev/cli install, and start with /gaffa-find on a site you need data from.

Originally published at kevinmeneses.com on November 6, 2026.

Web Scraping with Claude Code: A Working Workflow with Gaffa was originally published in CodeX on Medium, where people are continuing the conversation by highlighting and responding to this story.