When an incompatible connection is closed, clear the state that prevents us from sending messages on it

[sfc-gh-abeamon]

When downgrading a cluster, it was possible for a Peer object on the destination version to be held open through the downgrade. Before the downgrade, the Peer would have marked the connection as incompatible and would stop sending messages to it. After the downgrade, that state would persist and it would continue refusing to send messages even though the remote endpoint is compatible.

With this change, the state marking the connection as ineligible for messages is cleared when an incompatible connection is closed.

Passed 10K correctness. Also passes downgrade testing when applied to 7.0 and 7.1.

Code-Reviewer Section

The general guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• The PR has a description, explaining both the problem and the solution.
• The description mentions which forms of testing were done and the testing seems reasonable.
• Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
• There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: 0597ae7
• Result: ✔️ SUCCEEDED
• Build Logs (available for 30 days)

[foundationdb-ci]

AWS CodeBuild CI Report for macOS BigSur 11.5.2

• CodeBuild project: foundationdb-pr-macos
• Commit ID: 0597ae7
• Result: SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[foundationdb-ci]

AWS CodeBuild CI Report for Linux CentOS 7

• CodeBuild project: foundationdb-pr
• Commit ID: 0597ae7
• Result: SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)