Skip to content

Add workshop step 23: Test Your Prompt Ideas with A/B Experiments#983

Merged
pelikhan merged 3 commits into
mainfrom
copilot/training-plan-add-research-backup-node
Jul 14, 2026
Merged

Add workshop step 23: Test Your Prompt Ideas with A/B Experiments#983
pelikhan merged 3 commits into
mainfrom
copilot/training-plan-add-research-backup-node

Conversation

Copilot AI commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

The curriculum (steps 1–22) teaches building, scheduling, memory, sub-agents, and error handling — but never gives learners a principled way to measure whether a prompt change actually improves output. Step 23 closes that gap with the experiments: frontmatter field.

New file: workshop/23-ab-experiments.md

Covers the full experiment lifecycle:

  • How the round-robin works — state loaded from experiments/{workflow-id} branch, lowest-count variant picked, counters saved, artifact uploaded, variant injected before agent sees the prompt
  • Frontmatter syntaxexperiments: block with named experiment and variant array; naming rules ([a-zA-Z_][a-zA-Z0-9_]*, non-matching names silently skipped at compile time)
  • Task brief integration{{#if experiments.<name> == "<variant>" }} blocks to swap instructions per variant
experiments:
  output_style: [concise, detailed]
{{#if experiments.output_style == "concise" }}
Write a maximum of 5 bullet points. Each bullet is one sentence.
{{#else}}
Write a structured report with sections: open issues, merged PRs, CI status.
{{#endif}}
  • Inspecting results — downloading the experiment artifact JSON to check per-variant assignment counts
  • Concluding — promote winning variant to baseline, remove experiments: block, recompile; do not remove early or state resets

Updated: workshop/README.md

Added step 23 row to the curriculum table.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add research-backed training node for A/B experiments Add workshop step 23: Test Your Prompt Ideas with A/B Experiments Jul 14, 2026
Copilot AI requested a review from pelikhan July 14, 2026 09:57
@pelikhan

Copy link
Copy Markdown
Contributor

@copilot emphasize on using the /agentic-workflows skill to author experiments

…xperiments

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

@copilot emphasize on using the /agentic-workflows skill to author experiments

Done in the latest commit. The "Add an experiment to your workflow" section now leads with the agent path: a [!TIP] callout establishes agents edit agents as the preference, followed by a ready-to-paste Copilot prompt that instructs the agent to use the /agentic-workflows skill and handle the full edit-compile-commit cycle. The manual frontmatter editing steps are kept as a secondary "alternative" subsection.

@pelikhan pelikhan marked this pull request as ready for review July 14, 2026 10:08
@pelikhan pelikhan merged commit f27b9ec into main Jul 14, 2026
@pelikhan pelikhan deleted the copilot/training-plan-add-research-backup-node branch July 14, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[training-plan] Add research-backed training node: Test Your Prompt Ideas with A/B Experiments

2 participants