Redesigning the Test Team Updates Report

[SirLouen]

We need to redesign the following report:
https://make.wordpress.org/updates/tag/test/

We should start only taking those sections where the Test Team agency, and remove all that we don't have

The test shows:

1. Current totals per category
2. 2-week changes per category
3. Closed tickets per category.

The first two make sense, but the last one doesn't provide much value as the two first actually include this information. So basically I would stick only to the first 2 to keep things simpler.

Analysing current categories

Which should be the categories where Test Team have Agency?

1. Needs Testing Info: This is part of the triaging group, aka the component maintainers. Component maintainers should be accountant of which tickets have not received answer in long enough time to close or to rethink them (maybe if the patch is not clear, create a new ticket, and if the ticket is not clear with the patch, change it to reporter-feedback, and close it after a couple of weeks if there is no answer). So this report should be removed, for both Core and Gutenberg, from this report.

2. Needs to Reproduce Issue: This is important, both in Core and Gutenberg.

• Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+-has-patch&focuses=!docs&col=id&col=summary&col=type&col=focuses&col=keywords&col=owner&col=priority&col=changetime&order=changetime
  Characteristics:

  • It must not be a doc, as it's not testable
  • It must have needs-testing but not -has-patch. This was decided sone meetings ago,
  • All except those closed one.

• Gutenberg: https://github.com/WordPress/gutenberg/issues?q=state%3Aopen%20label%3A%22Needs%20Testing%22

  • This is simpler because Gutenberg goes over GitHub, and GitHub clearly differentiates Issues from Pull Requests

3. Needs Patch Testing: Similarly, this is one of the most important

• Core: The other way around from Needs Reproduction. needs-testing and has-patch: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+has-patch&focuses=!docs&col=id&col=summary&col=focuses&col=keywords&col=owner&col=type&col=priority&col=changetime&order=changetime
• Gutenberg: Needs Testing in the PRs: https://github.com/WordPress/gutenberg/pulls?q=is%3Aopen+is%3Apr+label%3A%22Needs+Testing%22

4. Need Unit Tests: Currently, we don't have any clear directives about Unit Testing and poor agency. At this moment, this is a Core Team task, mostly done by patch makers or committers. Maybe we need to rethink how to introduce this in the future, but not now.

5. Need Review: Like the first element, reviewing is one of the main tasks of the component maintainers (Triaging and Reviewing). So this should not be part of this report either.

Conclusion

My proposal is just to add For now only 2 sections with 4 reports each:

Current Totals

NOTE: This should account for changes between the last report.

1. Need to Reproduce Issue in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+-has-patch&focuses=!docs&col=id&col=summary&col=type&col=focuses&col=keywords&col=owner&col=priority&col=changetime&order=changetime
2. Need to Reproduce Issue in Editor: https://github.com/WordPress/gutenberg/issues?q=state%3Aopen%20label%3A%22Needs%20Testing%22
3. Needs Patch Testing in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+has-patch&focuses=!docs&col=id&col=summary&col=focuses&col=keywords&col=owner&col=type&col=priority&col=changetime&order=changetime
4. Needs Patch Testing in Editor: https://github.com/WordPress/gutenberg/pulls?q=is%3Aopen+is%3Apr+label%3A%22Needs+Testing%22

New/Changed from last report

NOTE: This doesn't need the (Change values), as this is only informative and not correlated among weeks.

1. Need to Reproduce Issue in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&changetime=2weeksago&focuses=!docs&keywords=~needs-testing+-has-patch&col=id&col=summary&col=focuses&col=keywords&col=owner&col=type&col=priority&col=changetime&order=changetime
2. Need to Reproduce Issue in Editor: https://github.com/WordPress/gutenberg/issues?q=state%3Aopen%20label%3A%22Needs%20Testing%22%20created%3A%3E%40today-14d
3. Needs Patch Testing in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+has-patch&focuses=!docs&col=id&col=summary&col=focuses&col=keywords&col=owner&col=type&col=priority&col=changetime&order=changetime&changetime=2weeksago
4. Needs Patch Testing in Editor https://github.com/WordPress/gutenberg/pulls?q=is%3Aopen+is%3Apr+label%3A%22Needs+Testing%22+created%3A%3E2025-12-01+
   NOTE: this report doesn't support the today-14d so it must be updated with the date that the last report was published. Remember that the format is YYYY-MM-DD

[Successfulsebunya]

I agree to this. Let me also add it in the week in test Post for visibility

[nikunj8866]

@SirLouen

Needs to Reproduce Issue:
Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+-has-patch&focuses=!docs&col=id&col=summary&col=type&col=focuses&col=keywords&col=owner&col=priority&col=changetime&order=changetime

This query also includes Enhancement type tickets. Is that fine for this report, or should it only include Bug tickets since reproduction is mainly required for bugs?

Suggested Query (Bugs Only):

https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+-has-patch&focuses=!docs&type=defect+(bug)&col=id&col=summary&col=type&col=focuses&col=keywords&col=owner&col=priority&col=changetime&order=changetime

[nikunj8866]

Needs Patch Testing in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+has-patch&focuses=!docs&col=id&col=summary&col=focuses&col=keywords&col=owner&col=type&col=priority&col=changetime&order=changetime

I would prefer the current report query, as it already filters out tickets with keywords like -dev-feedback, -2nd-opinion, -close, -needs-refresh, -needs-design, -needs-design-feedback, and -needs-test-info, which makes the list more relevant for patch testing.

Suggested Query (Patch Testing):
https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+has-patch+-dev-feedback+-2nd-opinion+-close+-needs-refresh+-needs-design+-needs-design-feedback+-needs-test-info&milestone=!Awaiting+Review&col=id&col=summary&col=status&col=milestone&col=keywords&col=type&col=priority&col=component&col=changetime&order=changetime

[SirLouen]

@nikunj8866 after commenting this with in the test team program, the target is to clear up 100% of the tickets with needs-testing, no more filtering down. From now on, the Test Team, will "own" the needs-testing tickets as it should be.

We will triage every single ticket, remove the needs-testing when it's wrong, and get all the reports down to zero with this label, just like we did with Gutenberg. Its not about having a list of tickets to test, but also, correcting the wrongly labeled tickets.

Basically now we are doing a "garbage collector activity" to remove needs-testing from Tickets that are wrongly tagged. I've also asked in Meta, to see if we can get alerts on #core-test when a new ticket is tagged as needs-testing in Core, so we can act quickly and remove the wrong tags when they are labeled wrongly.

In the first week of program we have dropped already a 10% of the tickets and I hope to get to 0 before summer. With the "Updates" report we will see the progress.

So we are going to be simplifying to the max the queries so we don't have to think if this or that should be considered a test eligible report. We will go through the middle path.

Let me put you an example: Imagine than a ticket has dev-feedback and needs-testing. We read through the ticket, and we see that the ticket is waiting for a Committer to say a verdict about the last patch sent. Should we test this? No. Then we should simply remove needs-testing and move on. The needs-testing tag was wrongly set. But what would happen if there was already a verdict by a Commiter and someone forgot to remove dev-feedback? Our current "query" will be hiding this ticket from our eyes to be tested. This is where we should be removing the dev-feedback label, and add the corresponding Test Report. This is why we should be as "clean" as possible and focus on the whole needs-testing label. needs-testing should be a label that Committers/Maintainers/Gardeners set when they really think that a ticket should be tested. Otherwise, we should clear up that label as soon as we see it.

Long Story Short

For Core:

• needs-testing and -has-patch = Needs Reproduction
• needs-testing and has-patch = Needs Patch Testing

For Gutenberg:

• Issues section and Needs Testing = Needs Reproduction
• Pull Requests section and Needs Testing = Needs Patch Testing

Other Github Repositories in the future:

• Same as Gutenberg.

Final Proposal:

Current Totals

NOTE: This should account for changes between the last report.

• Need to Reproduce Issue in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+-has-patch&col=id&col=summary&col=focuses&col=keywords&col=owner&col=type&col=priority&col=changetime&order=changetime
• Need to Reproduce Issue in Editor: https://github.com/WordPress/gutenberg/issues?q=state%3Aopen%20label%3A%22Needs%20Testing%22
• Needs Patch Testing in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&keywords=~needs-testing+has-patch&col=id&col=summary&col=focuses&col=keywords&col=owner&col=type&col=priority&col=changetime&order=changetime
• Needs Patch Testing in Editor: https://github.com/WordPress/gutenberg/pulls?q=is%3Aopen+is%3Apr+label%3A%22Needs+Testing%22

New/Changed from last report

NOTE: This doesn't need the (Change values), as this is only informative and not correlated among weeks.

• Need to Reproduce Issue in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&changetime=2weeksago..&keywords=~needs-testing+-has-patch&col=id&col=summary&col=changetime&col=focuses&col=keywords&col=owner&col=type&col=priority&order=changetime
• Need to Reproduce Issue in Editor: https://github.com/WordPress/gutenberg/issues?q=state%3Aopen%20label%3A%22Needs%20Testing%22%20created%3A%3E%40today-14d
• Needs Patch Testing in Core: https://core.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&changetime=2weeksago..&keywords=~needs-testing+has-patch&col=id&col=summary&col=changetime&col=focuses&col=keywords&col=owner&col=type&col=priority&order=changetime
• Needs Patch Testing in Editor https://github.com/WordPress/gutenberg/pulls?q=is%3Aopen+is%3Apr+label%3A%22Needs+Testing%22+created%3A%3E2025-12-01
  IMPORTANT NOTE: this report doesn't support the today-14d so it must be updated with the date that the last report was published. Remember that the format is YYYY-MM-DD (the date is in the end of the URL)

So these reports are pretty much ready to be used from next week onwards.

[nikunj8866]

@SirLouen What’s your view on how we should handle the milestone for has-patch tickets? Should we consider all tickets, or should we skip those that are currently in Awaiting Review status?

[SirLouen]

@nikunj8866 in that case we should evaluate if it really ready for testing or not
For example, say that someone at random has created a patch and added the tag
In that case, if we think that this needs first a review, then we remove the needs-testing and add the dev-feedback.

As I said, we need to act as a "garbage collector" and/or repair the wrongly tagged tickets, but we must "own", the needs-testing label completely.