Skip to content

Conversation

@blindpirate
Copy link
Contributor

@blindpirate blindpirate commented Mar 16, 2019

Describe what this PR does / why we need it

In my previous #574 , I triggered travis build 3 times to get a successful result. This is too painful. This PR introduces RetryRule to retry tests in order to mitigate the test flakiness.

This is not a perfect solution (which should be rewriting the tests), but it's better than none.

A more modern alternative would be switching to JUnit 5 and using https://github.com/artsok/rerunner-jupiter .

Does this pull request fix one issue?

NONE

Describe how you did it

Create a RetryRule, which reruns failed test cases.

Tests for RetryRule are also added.

Describe how to verify it

Run travis build many times. There should be no flakiness anymore.

Special notes for reviews

Note: During retry, Before methods will be invoked, but the test class instance won't be reset.

@codecov-io
Copy link

codecov-io commented Mar 16, 2019

Codecov Report

Merging #582 into master will increase coverage by 1.25%.
The diff coverage is 73.58%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #582      +/-   ##
============================================
+ Coverage     39.86%   41.12%   +1.25%     
+ Complexity     1244     1215      -29     
============================================
  Files           275      267       -8     
  Lines          8702     7852     -850     
  Branches       1163     1062     -101     
============================================
- Hits           3469     3229     -240     
+ Misses         4785     4230     -555     
+ Partials        448      393      -55
Impacted Files Coverage Δ Complexity Δ
...re/src/main/java/com/alibaba/csp/sentinel/Env.java 75% <ø> (-5%) 1 <0> (ø)
...ain/java/com/alibaba/csp/sentinel/log/LogBase.java 69.64% <0%> (ø) 10 <0> (ø) ⬇️
...va/com/alibaba/csp/sentinel/init/InitExecutor.java 13.15% <0%> (+0.33%) 2 <0> (ø) ⬇️
.../sentinel/slots/nodeselector/NodeSelectorSlot.java 94.73% <100%> (ø) 4 <0> (ø) ⬇️
...nel/annotation/aspectj/SentinelResourceAspect.java 84.21% <100%> (ø) 3 <0> (?)
...sentinel/adapter/reactor/FluxSentinelOperator.java 100% <100%> (ø) 2 <0> (ø) ⬇️
...l/adapter/dubbo/fallback/DefaultDubboFallback.java 100% <100%> (ø) 2 <2> (?)
...tinel/slots/clusterbuilder/ClusterBuilderSlot.java 71.42% <100%> (ø) 8 <0> (ø) ⬇️
...l/annotation/aspectj/ResourceMetadataRegistry.java 83.33% <100%> (ø) 10 <2> (?)
.../sentinel/adapter/dubbo/DubboAppContextFilter.java 100% <100%> (ø) 3 <3> (?)
... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cda4f7e...4a024d7. Read the comment docs.

*
* @author jialiang.linjl
*/
public class OccupiableBucketLeapArrayTest {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added @Retry to all test cases of this class since I observed flaky failures in travis build history. I assume this is a common issue of all test cases in this class.

@blindpirate
Copy link
Contributor Author

@sczyh30 sczyh30 added to-review To review area/test Issue or PR related to test cases labels Mar 16, 2019
Previously, we have been bitten by flaky tests. This commit introduces RetryRule
to mitigate flakiness.
@sczyh30
Copy link
Member

sczyh30 commented Mar 26, 2019

That's cool for auto-retrying unstable tests. Thanks! We might need to rewrite some of these tests later to be stable and time-independent (though it's tricky).

sczyh30 and others added 22 commits April 10, 2019 09:20
…baba#617)

because Dubbo 2.5.x will be deprecated soon

Signed-off-by: Eric Zhao <[email protected]>
… in LogBase (alibaba#613)

to avoid affecting normal logic of users

Signed-off-by: Eric Zhao <[email protected]>
- copied legacy code and adapts to new package
- also update document for legacy Dubbo adapter

Signed-off-by: Eric Zhao <[email protected]>
Signed-off-by: Eric Zhao <[email protected]>
luoxn28 and others added 6 commits April 10, 2019 09:20
@sczyh30 sczyh30 removed the to-review To review label May 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/test Issue or PR related to test cases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants