Fix flaky MemoryRecordsBuilderTest#testBuffersDereferencedOnClose

The test depends on GC to verify the memory usage. However, that is unstable. All we want to prove is that the inner "object" will drop the reference. Hence, we should rewrite the test to check the reference of inner object instead of calling GC.