Skip to content
This repository was archived by the owner on Mar 12, 2026. It is now read-only.

fix: avoid panic when the file is corrupted in disk cache#1130

Merged
ShiKaiWi merged 9 commits into
apache:mainfrom
ShiKaiWi:fix-avoid-massive-backtrace-output
Aug 8, 2023
Merged

fix: avoid panic when the file is corrupted in disk cache#1130
ShiKaiWi merged 9 commits into
apache:mainfrom
ShiKaiWi:fix-avoid-massive-backtrace-output

Conversation

@ShiKaiWi

@ShiKaiWi ShiKaiWi commented Aug 3, 2023

Copy link
Copy Markdown
Member

Rationale

  • The corrupted file in the disk cache will lead to the panic of the server
  • The whole cache page file is fetched when only a range of the file is accessed, leading to high memory consumption

Detailed Changes

  • Store the file size in the memory for file integrity check
  • Omit the corrupted file in the disk cache rather than panic
  • Do IO operation outside the meta data cache lock
  • Read only the involved range of a cached paged file instead of the whole file

Test Plan

Existing tests and a newly designed test.

@ShiKaiWi ShiKaiWi force-pushed the fix-avoid-massive-backtrace-output branch 4 times, most recently from 7da4af9 to ee8a5f3 Compare August 4, 2023 09:21
Comment thread components/object_store/src/disk_cache.rs
Comment thread components/object_store/src/disk_cache.rs
@ShiKaiWi ShiKaiWi force-pushed the fix-avoid-massive-backtrace-output branch 2 times, most recently from 0ad7674 to 3d1ef7b Compare August 7, 2023 08:37
@ShiKaiWi ShiKaiWi force-pushed the fix-avoid-massive-backtrace-output branch from 3d1ef7b to 3670f22 Compare August 8, 2023 06:14
Comment thread components/object_store/src/disk_cache.rs
Comment thread components/object_store/src/disk_cache.rs Outdated
Comment thread components/object_store/src/disk_cache.rs Outdated
Comment thread components/object_store/src/disk_cache.rs Outdated
Comment thread components/object_store/src/disk_cache.rs
Comment thread components/object_store/src/disk_cache.rs Outdated
Comment thread components/object_store/src/disk_cache.rs Outdated
Comment thread components/object_store/src/disk_cache.rs Outdated

@jiacai2050 jiacai2050 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShiKaiWi ShiKaiWi merged commit f5b2175 into apache:main Aug 8, 2023
@ShiKaiWi ShiKaiWi deleted the fix-avoid-massive-backtrace-output branch August 8, 2023 11:40
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants