[Perf] Linux/arm64: 7 Regressions on 3/28/2023 10:24:48 AM #14634
Description
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | ed3721b092e395ac3576c2d93f4134650a44dc08 |
| Compare | 49ae58ace038dbbd3d53147e08c9ac95b24a789f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| MatchesSet - Duration of single invocation | 65.93 μs | 77.51 μs | 1.18 | 0.08 | False | |||||
| OneNodeBacktracking - Duration of single invocation | 91.31 ns | 126.20 ns | 1.38 | 0.28 | False | |||||
| MatchesBoundary - Duration of single invocation | 61.22 μs | 85.85 μs | 1.40 | 0.05 | True | |||||
| MatchesSet - Duration of single invocation | 65.69 μs | 77.38 μs | 1.18 | 0.03 | True | |||||
| MatchesBoundary - Duration of single invocation | 61.41 μs | 85.95 μs | 1.40 | 0.02 | True | |||||
| Date_IsNotMatch - Duration of single invocation | 143.36 ns | 179.36 ns | 1.25 | 0.02 | True | |||||
| Date_IsNotMatch - Duration of single invocation | 139.84 ns | 178.91 ns | 1.28 | 0.02 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: IgnoreCase, Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 77.514115313214 > 69.49939937752309.
IsChangePoint: Marked as a change because one of 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -18.958138603491705 (T) = (0 -76947.1684807885) / Math.Sqrt((8496634.947970118 / (33)) + (130888.88992695129 / (6))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (6) - 2, .025) and -0.14969698318495897 = (66928.21639631069 - 76947.1684807885) / 66928.21639631069 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 126.19567320874324 > 100.74592776046136.
IsChangePoint: Marked as a change because one of 1/5/2023 10:03:03 PM, 1/15/2023 4:47:52 PM, 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -34.52804837981968 (T) = (0 -127.71426184138913) / Math.Sqrt((6.082417785131605 / (33)) + (4.41594097943025 / (6))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (6) - 2, .025) and -0.3501789748061439 = (94.59061666970936 - 127.71426184138913) / 94.59061666970936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 85.85032978479852 > 64.44862728514404.
IsChangePoint: Marked as a change because one of 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -164.05687266755143 (T) = (0 -85150.41426074681) / Math.Sqrt((18500.112642650867 / (33)) + (122396.74430169149 / (6))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (6) - 2, .025) and -0.38683873987671913 = (61398.92967535369 - 85150.41426074681) / 61398.92967535369 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 77.38079769230771 > 69.6607456010894.
IsChangePoint: Marked as a change because one of 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -56.68358683752159 (T) = (0 -76665.51084222078) / Math.Sqrt((252040.94123439136 / (32)) + (155802.63837915318 / (6))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (6) - 2, .025) and -0.15743060075355678 = (66237.67402754596 - 76665.51084222078) / 66237.67402754596 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: IgnoreCase, Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 85.95272614644969 > 64.44590341085687.
IsChangePoint: Marked as a change because one of 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -133.32827741160585 (T) = (0 -85127.8385115181) / Math.Sqrt((44515.818974660084 / (33)) + (182088.97935423913 / (6))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (6) - 2, .025) and -0.3866616741779852 = (61390.48918474075 - 85127.8385115181) / 61390.48918474075 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: IgnoreCase, Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 179.35817492139208 > 150.3545781010817.
IsChangePoint: Marked as a change because one of 1/27/2023 7:12:54 PM, 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -38.034303602809935 (T) = (0 -180.62935285729) / Math.Sqrt((4.8085696229625645 / (33)) + (5.379455708739078 / (6))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (6) - 2, .025) and -0.2738395681827048 = (141.79913810887575 - 180.62935285729) / 141.79913810887575 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 178.91159432850742 > 148.7727507789894.
IsChangePoint: Marked as a change because one of 1/27/2023 7:12:54 PM, 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -33.773539593410895 (T) = (0 -180.94239430066202) / Math.Sqrt((5.464994894812919 / (33)) + (7.048810836416668 / (6))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (6) - 2, .025) and -0.2756728672463426 = (141.840748475934 - 180.94239430066202) / 141.840748475934 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository