Re: Improving tracking/processing of buildfarm test failures

02.07.2024 15:00, Alexander Lakhin wrote:
>
> One month later,  I'd like to summarize failures that I've investigated
> and classified during June, 2024 on the aforementioned wiki page.
> (Maybe it would make sense to issue a monthly report with such information
> in the future.)

Please take a look at July report on the buildfarm failures:
# SELECT br, count(*) FROM failures WHERE dt >= '2024-07-01' AND
 dt < '2024-08-01' GROUP BY br;

REL_12_STABLE: 11
REL_13_STABLE: 9
REL_14_STABLE: 7
REL_15_STABLE: 10
REL_16_STABLE: 9
REL_17_STABLE: 68
HEAD: 106
-- Total: 220
(Counting test failures only, excluding indent-check, Configure, Build
errors.)

# SELECT COUNT(*) FROM (SELECT DISTINCT issue_link FROM failures WHERE
 dt >= '2024-07-01' AND dt < '2024-08-01');
40

# SELECT issue_link, count(*) FROM failures WHERE dt >= '2024-07-01' AND
 dt < '2024-08-01' GROUP BY issue_link ORDER BY 2 DESC LIMIT 9;

https://www.postgresql.org/message-id/[email protected]: 29
-- An environmental issue

https://www.postgresql.org/message-id/[email protected]: 20
-- Probably fixed

https://www.postgresql.org/message-id/[email protected]: 11
-- Fixed

https://www.postgresql.org/message-id/[email protected]: 9

https://www.postgresql.org/message-id/[email protected]: 8
-- An environmental issue; probably fixed

https://www.postgresql.org/message-id/[email protected]: 8

https://www.postgresql.org/message-id/[email protected]: 8
-- Fixed

https://www.postgresql.org/message-id/[email protected]: 8
-- Fixed

https://www.postgresql.org/message-id/[email protected]: 8
-- Fixed

# SELECT count(*) FROM failures WHERE dt >= '2024-07-01' AND
 dt < '2024-08-01' AND issue_link IS NULL; -- Unsorted/unhelpful failures
17

And one more metric, that might be useful, but it requires also time
analysis — short-lived (eliminated immediately) failures: 83

I also wrote a simple script (see attached) to check for unknown buildfarm
failures using "HTML API", to make sure no failures missed. Surely, it
could be improved in many ways, but I find it rather useful as-is.

Best regards,
Alexander