2026-11-15
Testing and Evaluating AI Systems in Ruby
Testing and Evaluating AI Systems in Ruby
Ruby developers building AI-powered applications face a distinct challenge: how do you test systems whose outputs are probabilistic and often generated by language models? Traditional testing approaches fall short when dealing with AI, where exact matches matter less than semantic correctness and consistent quality.
Several tools have emerged to help Ruby developers test and evaluate AI systems. Understanding what each offers will help you choose the right fit for your workflow.
RSpec-Based AI Testing
rspec-ai-formatter takes your existing RSpec test suite and enhances it with AI-driven analysis. When tests fail, this formatter uses AI to help you understand what went wrong faster. This is useful if you already rely on RSpec and want to extend it without major changes to your testing structure.
rspec-llm goes further by adding LLM-specific matchers and assertions directly into RSpec. Instead of testing for exact string matches, you can write assertions that verify AI outputs semantically. Use this when you want RSpec's familiar syntax but need matchers tailored for language model outputs.
Validation and Testing Frameworks
probatio_diabolica provides testing and validation utilities for AI-driven applications. It focuses on robust validation of AI system inputs and outputs, helping ensure your application handles edge cases properly.
roast, from Shopify, offers a framework for building and testing AI-powered features with structured outputs and validation. This is particularly valuable if you need to enforce specific output formats from language models and want a structured approach to testing them.
Prompt-Specific Testing
minitest-promptfoo integrates Promptfoo into Minitest, letting you test and evaluate prompts within your standard Ruby test suite. Use this if you rely on Minitest and need to validate prompt quality alongside your application code.
completion-kit is a dedicated framework for testing AI prompts. It lets you run prompts against datasets, score outputs using LLM judges, version experiments, and compare results. Choose this for systematic prompt evaluation outside your main test suite, especially if you iterate on prompts frequently.
Evaluation and Monitoring
braintrust provides tools for evaluating, testing, and monitoring language model applications. It enables systematic evaluation of your AI outputs against defined metrics, useful for tracking quality over time.
amg-harness is a regression and optimization harness specifically for LLM agents. It uses a deliberately blind judge to evaluate changes, helping you test agent behavior in real-world scenarios while measuring incremental improvements objectively.
Which should you choose?
Start by considering your testing framework. If you use RSpec, rspec-llm or rspec-ai-formatter keep testing integrated into your existing workflow. If Minitest is your preference, minitest-promptfoo works similarly.
For prompt-focused work, completion-kit is purpose-built for iteration and comparison. For agent testing, amg-harness offers real-world evaluation patterns.
If you need general validation for AI outputs, probatio_diabolica or roast provide structured approaches. For monitoring quality across runs, braintrust gives you systematic evaluation tools.
Many teams use multiple tools: a test framework for unit-level AI testing, a prompt evaluation tool for iteration, and a monitoring solution for production quality.