![about flakes
“... In terms of severity, of the 91% of developers who claimed to deal
with flaky tests at least a few times a year,
… 23% [of developers] thought that they were
a serious problem. …”
source: “A survey of flaky tests”](https://image.slidesharecdn.com/danielhiller-250506091253-bb889c12/85/stackconf-2025-Zap-the-Flakes-Leveraging-AI-to-Combat-Flaky-Tests-with-CANNIER-by-Daniel-Hiller-pdf-6-320.jpg)
![about flakes
“… test flakiness was a frequently encountered problem,
with … 15% [of developers] dealing with it daily”
source: “A survey of flaky tests”](https://image.slidesharecdn.com/danielhiller-250506091253-bb889c12/85/stackconf-2025-Zap-the-Flakes-Leveraging-AI-to-Combat-Flaky-Tests-with-CANNIER-by-Daniel-Hiller-pdf-7-320.jpg)
![CANNIER
“... we found that CANNIER was able to reduce the time cost (and therefore
monetary cost) [of re-running tests] by an average of 88% ...”
source: https://www.gregorykapfhammer.com/research/papers/parry2023/](https://image.slidesharecdn.com/danielhiller-250506091253-bb889c12/85/stackconf-2025-Zap-the-Flakes-Leveraging-AI-to-Combat-Flaky-Tests-with-CANNIER-by-Daniel-Hiller-pdf-17-320.jpg)
stackconf 2025 | Zap the Flakes! Leveraging AI to Combat Flaky Tests with CANNIER by Daniel Hiller.pdf
0 likes39 views
Flakes aka tests that don’t behave deterministically, i.e. they fail sometimes and pass sometimes, are an ever recurring problem in software development. This is especially the sad reality when running end-to-end tests where a lot of components are involved. There are various reasons why a test can be flaky, however the impact can be as fatal as CI being loaded beyond capacity causing overly long feedback cycles or even users losing trust in CI itself. Ideally CI maintainers want potential flakes to be flagged at the earliest stage of the development lifecycle, so that they do not even enter the end-to-end test suite. They want a gate that acts as a safeguard for developers, pointing out to them what kind of stability issues a test has. This reduces CI user frustration and improves trust in CI. At the same time it cuts down on unnecessary waste of CI system resources. This talk will explore the CANNIER approach, which aims to reduce the time cost of rerunning tests. The speaker and the audience will take a closer look at the feature set used to create training data from existing test code of the KubeVirt project, enabling CI systems to predict probable flakiness of a certain test with an AI model. The speaker will cover how a subset of the CANNIER feature set is implemented, how it is used to generate training data for an AI model, and how the model is used to analyze flakiness probability, generating actionable feedback for test authors. Attendees will gain insight on how the CANNIER approach can help them improve test reliability, ultimately enhancing overall software quality and team productivity.
![[OrangeCon 2025 Keynote] Geeks to Giants: The Journey from Hacking Subculture...](https://cdn.slidesharecdn.com/ss_thumbnails/geeks2giantsorangecon25-250906081005-6768806b-thumbnail.jpg?width=560&fit=bounds)
stackconf 2025 | Zap the Flakes! Leveraging AI to Combat Flaky Tests with CANNIER by Daniel Hiller.pdf
- 1. Zap the Flakes! LeveragingAI to Combat Flaky Tests with CANNIER stackconf ‘25 Daniel Hiller
- 2. agenda ● about me ●about flakes ● pre-merge-detection v1 ● CANNIER ● pre-merge-detection v2 ● Q&A
- 3. about me ● SoftwareEngineer | Red Hat OpenShift Virtualization ● KubeVirt | CI & automation in general ● prow.ci.kubevirt.io ● CI-Health ● Flaky Tests kubevirt.io- Virtualization for Kubernetes
- 4.
- 5. about flakes a flake isa test that without any code change will either fail or pass in successive runs
- 6. about flakes “... Interms of severity, of the 91% of developers who claimed to deal with flaky tests at least a few times a year, … 23% [of developers] thought that they were a serious problem. …” source: “A survey of flaky tests”
- 7. about flakes “… testflakiness was a frequently encountered problem, with … 15% [of developers] dealing with it daily” source: “A survey of flaky tests”
- 8.
- 9. in CI automatedtesting MUST give a reliable signal of stability any failed test run signals that the product is unstable test runs failed due to flakes do not give this reliable signal they only waste time impact of flakes
- 10. impact of flakes Flakytests cause ● for individual contributors ○ prolonged feedback cycles ○ test trust issues ● for the project community ○ slowdown of merging pull requests - “retest trap” ○ reversal of acceleration effects (i.e. batch testing) ○ waste of CI resources
- 11. pre-merge-detection v1 approach: ● lookat files touched in pull request ● determine changed files containing e2e tests ● run these 5 times in random order sources: KubeVirt e2e test runtimes, check-tests-for-flakes job history, automation/repeated_test.sh
- 12. pre-merge-detection v1 check-tests-for-flakes testlane why: catch flakes before entering main (source) ●
- 13. pre-merge-detection v1 why: ● eliminateorder-dependencies ● running five times is the boundary to find 88% of flaky tests sources: KubeVirt e2e test runtimes, check-tests-for-flakes job history, automation/repeated_test.sh
- 14. pre-merge-detection v1 initial situation: ●most e2e tests take 10sec - 2mins to run ● 5 times re-run has “only” 88% chance of detection sources: KubeVirt e2e test runtimes, check-tests-for-flakes job history, automation/repeated_test.sh
- 15. pre-merge-detection v1 problems discoveredwith v1 approach: ● shotgun approach causes too many tests to be run ○ capping amount of tests re-run required to avoid timeouts ○ CI users complain about failing tests they didn’t touch ● re-run lane takes ~1h on average sources: KubeVirt e2e test runtimes, check-tests-for-flakes job history, automation/repeated_test.sh
- 16. CANNIER CANNIER is anapproach for reducing the time cost of rerunning-based detection techniques by combining them with machine learning models source: https://www.gregorykapfhammer.com/research/papers/parry2023/
- 17. CANNIER “... we foundthat CANNIER was able to reduce the time cost (and therefore monetary cost) [of re-running tests] by an average of 88% ...” source: https://www.gregorykapfhammer.com/research/papers/parry2023/
- 18.
- 19.
- 20.
- 21.
- 22. pre-merge-detection v2.0 ● CANNIER ○implemented in Python ○ KubeVirt uses Go ● Runtime Data ○ where to store ○ when to capture ● data science ○ Python has well known frameworks ○ Go state unsure questions
- 23. pre-merge-detection v2.0 steps: ● gatherthe set of changed tests ● per each element ○ run once to get the runtime data ○ extract the static analysis data ○ fetch the prediction using entire feature vector ○ add to the set of re-run tests if in iDFClass ● re-run the reduced set of tests source: https://www.gregorykapfhammer.com/research/papers/parry2023/ algorithm
- 24. pre-merge-detection v2.0 parts: ● code(Go) ○ test set extraction ○ feature extraction ○ model prediction ○ model generation ● test lane (Bash) ● model deployment (YAML) source: https://www.gregorykapfhammer.com/research/papers/parry2023/ implementation
- 25. pre-merge-detection v2.1 parts: ● model ○generation ○ deployment ● test lane ● code ○ test set extraction ○ feature extraction ○ model prediction parts: ● model ○ … ○ automatic updates ● test lane -> prow external-plugin ○ runs on presubmits ○ runs on postcommits (probably with a larger test set) ○ adds helpful feedback ● code ○ test set extraction ○ feature extraction ○ model prediction improvements
- 26. pre-merge-detection v2.1 prow external-plugin ●runs on presubmits ● adds helpful feedback ● gives advice according to the feature set what can be improved
- 27. pre-merge-detection v2.2 ● componentsof feature vector provide insightful advice to the contributor ○ i.e. high cyclomatic complexity advises to reduction etc. ○ therefore it’s valuable to attach the analysis data to the PR ● possibly increase number of reruns (>5) ○ reduced runtime overall leaves time for more reruns ○ feature vector contains runtimes, thus we can estimate the total re-run time better and optimize for it, i.e. group tests by runtime classes source: https://www.gregorykapfhammer.com/research/papers/parry2023/ improvements
- 28. pre-merge-detection v1.5 (thepresent) approach: ● gather the exact set of changed tests ● run these 5 times in random order sources: KubeVirt e2e test runtimes, check-tests-for-flakes job history, automation/repeated_test.sh
- 29. Links ● KubeVirt resources ○Initial pull request (draft): https://github.com/kubevirt/project-infra/pull/3930 ○ Presentation Squash The Flakes @ FOSDEM ‘24: https://archive.fosdem.org/2024/schedule/event/fosdem-2024-1805-squash-the-flakes-ho w-to-minimize-the-impact-of-flaky-tests/ ● CANNIER ○ paper: https://www.gregorykapfhammer.com/research/papers/parry2023/ ○ implementation: https://github.com/flake-it/cannier-framework/
- 30. Q&A Any questions? Any suggestionsfor improvement? Who else is trying to tackle this problem? What have you done to solve this? download slides: slides.pdf
- 31. Thank you forattending! Further questions? Feel free to send questions and comments to: �� [email protected] kubernetes.slack.com/ @dhiller @[email protected] �� dhiller.dev kubevirt.io Virtualization for Kubernetes KubeVirt welcomes all kinds of contributions! ● Weekly community meeting every Wed 3PM CET ● Links: ● KubeVirt website ● KubeVirt user guide ● KubeVirt Contribution Guide ● GitHub ● Kubernetes Slack channels ○ #virtualization ○ #kubevirt-dev contact me: