[improve][broker]Improve the log when encountered in-flight read limitation

[poorbarcode]

Motivation & Modifications

• When reaching the pending read queue size limitation
  • Highlight the config managedLedgerMaxReadsInFlightPermitsAcquireQueueSize when the broker reaches the limitation that the pending read queue is full.
  • Instead of printing an error-level log, print a warning-level log.
• Additional print queue size and the limit threshold when the pending read queue is full
• Print warning-level log if reached in-flight messages size is reached, instead of debug-level.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[poorbarcode]

/pulsarbot rerun-failure-checks

[lhotari]

This tuning recommendation about increasing managedLedgerMaxReadsInFlightPermitsAcquireQueueSize isn't so straight forward since managedLedgerMaxReadsInFlightSizeInMB and managedLedgerMaxReadsInFlightPermitsAcquireTimeoutMillis settings also impact the situation.

If the managedLedgerMaxReadsInFlightSizeInMB value is higher, the queuing wouldn't happen in the first place. The general tuning recommendation should be to primarily increase managedLedgerMaxReadsInFlightSizeInMB, make more direct memory for the broker and possibly also increase the queue size.

When the timeout managedLedgerMaxReadsInFlightPermitsAcquireTimeoutMillis is shorter, the queue length would eventually be shorter when reads are rejected due to being in the queue for too long. However, reducing the timeout isn't very helpful since retries would happen eventually and that would just add unnecessary load to the system.

[lhotari]

LGTM

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.

Project coverage is 74.27%. Comparing base (bbc6224) to head (0175812).
Report is 1126 commits behind head on master.

Files with missing lines	Patch %	Lines
...keeper/mledger/impl/cache/RangeEntryCacheImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24359      +/-   ##
============================================
+ Coverage     73.57%   74.27%   +0.69%     
+ Complexity    32624    32284     -340     
============================================
  Files          1877     1866      -11     
  Lines        139502   145145    +5643     
  Branches      15299    16596    +1297     
============================================
+ Hits         102638   107802    +5164     
+ Misses        28908    28814      -94     
- Partials       7956     8529     +573     

Flag	Coverage Δ
inttests	26.68% <0.00%> (+2.10%)	⬆️
systests	23.32% <0.00%> (-1.01%)	⬇️
unittests	73.76% <87.50%> (+0.91%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...eeper/mledger/impl/cache/InflightReadsLimiter.java	85.03% <100.00%> (+2.89%)	⬆️
...keeper/mledger/impl/cache/RangeEntryCacheImpl.java	63.86% <0.00%> (+5.11%)	⬆️

... and 1081 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.