Skip to content

bolt: Improve delete inactive orders/matches perf.#3059

Merged
buck54321 merged 1 commit into
decred:masterfrom
jholdstock:rm-archived-perf
Nov 7, 2024
Merged

bolt: Improve delete inactive orders/matches perf.#3059
buck54321 merged 1 commit into
decred:masterfrom
jholdstock:rm-archived-perf

Conversation

@jholdstock

Copy link
Copy Markdown
Member

Running these functions on a resource contrained machine and a large database was incredibly slow due to overuse of the newestBuckets func. Every key was accessed and sorted every time a batch of orders/matches was processed.

Rather than repeatedly performing this work, the full set of keys is now accessed only once. The resulting code runs ~200x faster.

There is a slight change in behaviour because database transactions are now committed with a maximum of 1000 deletions rather than exactly 1000. This is not an issue because the exact size of each transaction doesn't matter, the important point is to limit the maximum size of each transaction to prevent excess memory consumption.

This performance gain removes the need to log the progress of each individual batch, so now only a summary is logged after the entire deletion process is completed.

Running these functions on a resource contrained machine and a large
database was incredibly slow due to overuse of the newestBuckets func.
Every key was accessed and sorted every time a batch of orders/matches
was processed.

Rather than repeatedly performing this work, the full set of keys is now
accessed only once. The resulting code runs ~200x faster.

There is a slight change in behaviour because database transactions are
now committed with a *maximum* of 1000 deletions rather than *exactly*
1000. This is not an issue because the exact size of each transaction
doesn't matter, the important point is to limit the maximum size of each
transaction to prevent excess memory consumption.

This performance gain removes the need to log the progress of each
individual batch, so now only a summary is logged after the entire
deletion process is completed.
@jholdstock

Copy link
Copy Markdown
Member Author

Just reviewed history and noticed @JoeGruffins was the original author of this code. Care to give it a once over Joe?

@JoeGruffins JoeGruffins left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't remember writing this but changes look good and tests passing.

@buck54321
buck54321 merged commit 35fe334 into decred:master Nov 7, 2024
buck54321 pushed a commit to buck54321/dcrdex that referenced this pull request Nov 12, 2024
Running these functions on a resource contrained machine and a large
database was incredibly slow due to overuse of the newestBuckets func.
Every key was accessed and sorted every time a batch of orders/matches
was processed.

Rather than repeatedly performing this work, the full set of keys is now
accessed only once. The resulting code runs ~200x faster.

There is a slight change in behaviour because database transactions are
now committed with a *maximum* of 1000 deletions rather than *exactly*
1000. This is not an issue because the exact size of each transaction
doesn't matter, the important point is to limit the maximum size of each
transaction to prevent excess memory consumption.

This performance gain removes the need to log the progress of each
individual batch, so now only a summary is logged after the entire
deletion process is completed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants