-
Notifications
You must be signed in to change notification settings - Fork 8.2k
Use RetryRule to mitigate test flakiness #582
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
@@ Coverage Diff @@
## master #582 +/- ##
============================================
+ Coverage 39.86% 41.12% +1.25%
+ Complexity 1244 1215 -29
============================================
Files 275 267 -8
Lines 8702 7852 -850
Branches 1163 1062 -101
============================================
- Hits 3469 3229 -240
+ Misses 4785 4230 -555
+ Partials 448 393 -55
Continue to review full report at Codecov.
|
| * | ||
| * @author jialiang.linjl | ||
| */ | ||
| public class OccupiableBucketLeapArrayTest { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added @Retry to all test cases of this class since I observed flaky failures in travis build history. I assume this is a common issue of all test cases in this class.
|
Hmmmm... seems not work well: https://travis-ci.org/alibaba/Sentinel/jobs/507205057 |
Previously, we have been bitten by flaky tests. This commit introduces RetryRule to mitigate flakiness.
|
That's cool for auto-retrying unstable tests. Thanks! We might need to rewrite some of these tests later to be stable and time-independent (though it's tricky). |
Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Carpenter Lee <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
…notation support Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
…baba#617) because Dubbo 2.5.x will be deprecated soon Signed-off-by: Eric Zhao <[email protected]>
… in LogBase (alibaba#613) to avoid affecting normal logic of users Signed-off-by: Eric Zhao <[email protected]>
- copied legacy code and adapts to new package - also update document for legacy Dubbo adapter Signed-off-by: Eric Zhao <[email protected]>
…sion Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
add blog address
Signed-off-by: Eric Zhao <[email protected]>
…ected entry type Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
…s negative (alibaba#642) - Also rearrangement of some code Signed-off-by: Eric Zhao <[email protected]>
* Add unit test for com.alibaba.csp.sentinel.util.IdUtil * Add unit tests for com.alibaba.csp.sentinel.util.StringUtil
Describe what this PR does / why we need it
In my previous #574 , I triggered travis build 3 times to get a successful result. This is too painful. This PR introduces
RetryRuleto retry tests in order to mitigate the test flakiness.This is not a perfect solution (which should be rewriting the tests), but it's better than none.
A more modern alternative would be switching to JUnit 5 and using https://github.com/artsok/rerunner-jupiter .
Does this pull request fix one issue?
NONE
Describe how you did it
Create a
RetryRule, which reruns failed test cases.Tests for
RetryRuleare also added.Describe how to verify it
Run travis build many times. There should be no flakiness anymore.
Special notes for reviews
Note: During retry,
Beforemethods will be invoked, but the test class instance won't be reset.