![\"\"](\"https://wanghenshui.github.io/assets/bytedance-tsdb1.png\")
省流 不析构，直接退出进程
\n\n\n\n\n\n他们的配置
\n\nrocksdb::LRUCacheOptions lru_block_cache_opts;\nlru_block_cache_opts.capacity = 1024*1024*1024*256; // 256 GiB\nlru_block_cache_opts.strict_capacity_limit = false;\nlru_block_cache_opts.high_pri_pool_ratio = 0.5;\nblock_cache_ = rocksdb::NewLRUCache(lru_block_cache_opts)\n内存256G 主要都卡在析构函数上了
\n\n我一开始给的点子是WAL或者主动停compaction之类的
\n\nblockcache析构一时没有好的点子
\n\nMark Callaghan给了个点子，直接跳过析构就好了
\n\n我怎么就没想到呢，脑子思维卡住了
\n\nrocksdb提供了跳过析构的接口
\n\n// Call this on shutdown if you want to speed it up. Cache will disown\n// any underlying data and will not free it on delete. This call will leak\n// memory - call this only if you're shutting down the process.\n// Any attempts of using cache after this call will fail terribly.\n// Always delete the DB object before calling this method!\nvirtual void DisownData() {\n// default implementation is noop\n}\n主动停止的时候主动调用一下就好了
\n\n我顺便蹭了一个PR https://github.com/apache/kvrocks/pull/2683
\n\n", "date_published": "Fri, 06 Dec 2024 00:00:00 +0000" }, { "id": "https://wanghenshui.github.io/2024/10/17/german-string.html", "url": "https://wanghenshui.github.io/2024/10/17/german-string.html", "title": "german string", "content_html": "prefix计算，空间换时间
\n\nhttps://cedardb.com/blog/german_strings/
\n\nhttps://cedardb.com/blog/strings_deep_dive/
\n\n这里有来源介绍
\n\numbra论文的来的，主要是为了小字符串 12B 做优化
\n\n\n\narrow
\n\n* Short strings, length <= 12\n | Bytes 0-3 | Bytes 4-15 |\n |------------|---------------------------------------|\n | length | data (padded with 0) |\n\n* Long strings, length > 12\n | Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 |\n |------------|------------|------------|-------------|\n | length | prefix | buf. index | offset |\nbuf index表示第几个buffer \nbuf内的[offset,offset+length) 表示字符串
\n\nhttps://pola.rs/posts/polars-string-type/
\n\nhttps://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout
\n\n还有一种常规设计
\n\n* Short strings, length <= 12\n | Bytes 0-3 | Bytes 4-15 |\n |------------|---------------------------------------|\n | length | data (padded with 0) |\n\n* Long strings, length > 12\n | Bytes 0-3 | Bytes 4-7 | Bytes 8-15 |\n |------------|------------|------------|\n | length | prefix | ptr |\n比较直观
\nbool isEqual(data128_t a, data128_t b) {\n if (a.v[0] != b.v[0]) return false;\n auto len = (uint32_t) a.v[0];\n if (len <= 12) return a.v[1] == b.v[1];\n return memcmp((char*) a.v[1], (char*) b.v[1], len) == 0;\n}\n对于小字符串 能省非常多
\n\n不过std::string也有sso，对于小字符串，但是没有prefix优势/16B传参优势
\n\nduckdb实现 duckdb/blob/main/src/include/duckdb/common/types/string_type.hpp
\n", "date_published": "Thu, 17 Oct 2024 00:00:00 +0000" }, { "id": "https://wanghenshui.github.io/2024/10/16/gen-memtier-csv.html", "url": "https://wanghenshui.github.io/2024/10/16/gen-memtier-csv.html", "title": "生成memtier 验证数据", "content_html": "用chatgpt生成的代码改的，chatgpt太强了
\n\n\n\n\n\nNote: \n To avoid unnecessary error while passing file.\n 1. Add extra 4 bytes in nbytes\n 2. Add extra 2 bytes in nkey\n \n Example:\n dumpflags, time, exptime, nbytes, nsuffix, it_flags, clsid, nkey, key, data\n 0, 0, 60, 19, 200, 0, 1, 11, 'doxrpshny', 'asdfghjklqwerty'\n主要是格式，以及多特殊预留位，以及引号
\n\n直接贴代码了
\n\n\nimport csv\nimport time\nimport random\nimport string\nimport argparse\nfrom typing import Generator, Dict\n\nclass RecordGenerator:\n \"\"\"记录生成器类\"\"\"\n def __init__(self):\n self.current_time = int(time.time())\n\n def generate_random_string(self, min_length: int = 5, max_length: int = 20) -> str:\n \"\"\"\n 生成随机字符串\n \"\"\"\n length = random.randint(min_length, max_length)\n res = ''.join(random.choices(string.ascii_lowercase, k=length))\n return \"'\" + res + \"'\"\n\n def generate_record(self) -> Dict:\n \"\"\"\n 生成单条记录\n \"\"\"\n # 生成随机key和data\n key = self.generate_random_string(20, 50)\n data = self.generate_random_string(60, 200)\n \n # 计算实际长度\n actual_key_length = len(key)\n actual_data_length = len(data)\n # To avoid unnecessary error while passing file \n # 1. Add extra 4 bytes in nbytes\n # 2. Add extra 2 bytes in nkey\n # 我们生成的数据是多引号的，已经+2了\n return {\n 'dumpflags': '0',\n ' time': ' 0', #self.current_time,\n ' exptime': ' 60', #self.current_time + random.randint(60, 3600),\n ' nbytes': f' {actual_data_length +2}',\n ' nsuffix': f' {random.randint(100, 300)}',\n ' it_flags': ' 0',\n ' clsid': ' 1',\n ' nkey': f' {actual_key_length}', ## 已经加2了\n ' key': f' {key}', \n ' data': f' {data}' \n }\n\n def records(self, num_records: int) -> Generator[Dict, None, None]:\n \"\"\"\n 生成记录的生成器\n \"\"\"\n for _ in range(num_records):\n yield self.generate_record()\n\nclass CSVWriter:\n \"\"\"CSV文件写入器类\"\"\"\n def __init__(self, filename: str):\n self.filename = filename\n self.fieldnames = [\n 'dumpflags', ' time', ' exptime', ' nbytes', ' nsuffix',\n ' it_flags', ' clsid', ' nkey', ' key', ' data'\n ]\n\n def write_records(self, records_generator: Generator[Dict, None, None]) -> int:\n \"\"\"\n 将记录写入CSV文件\n 返回写入的记录数量\n \"\"\"\n count = 0\n try:\n with open(self.filename, 'w', newline='') as csvfile:\n writer = csv.DictWriter(csvfile, fieldnames=self.fieldnames)\n writer.writeheader()\n \n for record in records_generator:\n writer.writerow(record)\n count += 1\n # 每1000条记录显示一次进度\n if count % 1000 == 0:\n print(f\"已生成 {count} 条记录...\")\n \n return count\n except Exception as e:\n print(f\"写入文件时发生错误：{e}\")\n return count\n\ndef parse_arguments() -> argparse.Namespace:\n \"\"\"解析命令行参数\"\"\"\n parser = argparse.ArgumentParser(description='memtierbenchmark data load CSV文件')\n parser.add_argument('-n', '--number', \n type=int, \n default=10,\n help='要生成的记录数量（默认：10）')\n parser.add_argument('-o', '--output',\n type=str,\n default='output.csv',\n help='output.csv）')\n return parser.parse_args()\n\ndef main():\n # 解析命令行参数\n args = parse_arguments()\n \n try:\n # 验证输入参数\n if args.number <= 0:\n raise ValueError(\"记录数量必须大于0\")\n \n print(f\"开始生成 {args.number} 条记录...\")\n start_time = time.time()\n\n # 创建生成器和写入器\n generator = RecordGenerator()\n writer = CSVWriter(args.output)\n \n # 生成并写入记录\n records_count = writer.write_records(generator.records(args.number))\n \n end_time = time.time()\n duration = end_time - start_time\n \n # 输出统计信息\n print(f\"\\n完成！\")\n print(f\"生成文件：{args.output}\")\n print(f\"记录数量：{records_count}\")\n print(f\"耗时：{duration:.2f} 秒\")\n print(f\"平均速度：{records_count/duration:.2f} 记录/秒\")\n \n except ValueError as ve:\n print(f\"参数错误：{ve}\")\n except Exception as e:\n print(f\"程序执行出错：{e}\")\n\nif __name__ == \"__main__\":\n main()\n\n# python3 gen.py -n 10000000 -o output.csv\n抄自trpc
\n\n\n\ninline void InitSpdLog() {\n bool truncate = true;\n auto max_files = 5;\n auto log_name = \"xxxlog_dir\" + \"/\" + \"INFO.%Y-%m-%d.log\";\n auto default_logger =\n spdlog::daily_logger_format_mt(\"xxxlog\", log_name, 12, 0, truncate, max_files);\n default_logger->set_level(spdlog::level::info);\n}\n\n\n#define MODULE_NAME_FMT(instance, level, formats, ...) \\\n do { \\\n spdlog::get(instance)->level(formats, ##__VA_ARGS__); \\\n } while (0)\n\n#define MODULE_NAME_IF(instance, condition, level, formats, ...) \\\n if (condition) { \\\n LOG_FMT(instance, level, formats, ##__VA_ARGS__); \\\n }\n\n#define LOGGER_FMT_TRACE(instance, format, args...) MODULE_NAME_FMT(instance, trace, format, ##args)\n#define LOGGER_FMT_DEBUG(instance, format, args...) MODULE_NAME_FMT(instance, debug, format, ##args)\n#define LOGGER_FMT_INFO(instance, format, args...) MODULE_NAME_FMT(instance, info, format, ##args)\n#define LOGGER_FMT_WARN(instance, format, args...) MODULE_NAME_FMT(instance, warn, format, ##args)\n#define LOGGER_FMT_ERROR(instance, format, args...) MODULE_NAME_FMT(instance, error, format, ##args)\n#define LOGGER_FMT_CRITICAL(instance, format, args...) MODULE_NAME_FMT(instance, critical, format, ##args)\n\n#define MODULE_NAME_TRACE(format, args...) LOGGER_FMT_TRACE(FLAGS_default_logger, format, ##args)\n#define MODULE_NAME_DEBUG(format, args...) LOGGER_FMT_DEBUG(FLAGS_default_logger, format, ##args)\n#define MODULE_NAME_INFO(format, args...) LOGGER_FMT_INFO(FLAGS_default_logger, format, ##args)\n#define MODULE_NAME_WARN(format, args...) LOGGER_FMT_WARN(FLAGS_default_logger, format, ##args)\n#define MODULE_NAME_ERROR(format, args...) LOGGER_FMT_ERROR(FLAGS_default_logger, format, ##args)\n#define MODULE_NAME_CRITICAL(format, args...) LOGGER_FMT_CRITICAL(FLAGS_default_logger, format, ##args)\n\n#define LOGGER_FMT_TRACE_IF(instance, condition, format, args...) \\\n MODULE_NAME_IF(instance, condition, trace, format, ##args)\n#define LOGGER_FMT_DEBUG_IF(instance, condition, format, args...) \\\n MODULE_NAME_IF(instance, condition, debug, format, ##args)\n#define LOGGER_FMT_INFO_IF(instance, condition, format, args...) \\\n MODULE_NAME_IF(instance, condition, info, format, ##args)\n#define LOGGER_FMT_WARN_IF(instance, condition, format, args...) \\\n MODULE_NAME_IF(instance, condition, warn, format, ##args)\n#define LOGGER_FMT_ERROR_IF(instance, condition, format, args...) \\\n MODULE_NAME_IF(instance, condition, error, format, ##args)\n#define LOGGER_FMT_CRITICAL_IF(instance, condition, format, args...) \\\n MODULE_NAME_IF(instance, condition, critical, format, ##args)\n\n#define MODULE_NAME_TRACE_IF(condition, format, args...) \\\n LOGGER_FMT_TRACE_IF(FLAGS_default_logger, condition, format, ##args)\n#define MODULE_NAME_DEBUG_IF(condition, format, args...) \\\n LOGGER_FMT_DEBUG_IF(FLAGS_default_logger, condition, format, ##args)\n#define MODULE_NAME_INFO_IF(condition, format, args...) \\\n LOGGER_FMT_INFO_IF(FLAGS_default_logger, condition, format, ##args)\n#define MODULE_NAME_WARN_IF(condition, format, args...) \\\n LOGGER_FMT_WARN_IF(FLAGS_default_logger, condition, format, ##args)\n#define MODULE_NAME_ERROR_IF(condition, format, args...) \\\n LOGGER_FMT_ERROR_IF(FLAGS_default_logger, condition, format, ##args)\n#define MODULE_NAME_CRITICAL_IF(condition, format, args...) \\\n LOGGER_FMT_CRITICAL_IF(FLAGS_default_logger, condition, format, ##args)\n\n#define MODULE_NAME_ASSERT(x) \\\n do { \\\n if (__builtin_expect(!(x), 0)) { \\\n [&]() __attribute__((noinline, cold)) { \\\n MODULE_NAME_CRITICAL(\"assertion failed: {}\", #x); \\\n std::abort(); \\\n } \\\n (); \\\n __builtin_unreachable(); \\\n } \\\n } while (0)\n\n#define MODULE_NAME_ONCE() \\\n ({ \\\n static std::atomic_flag __MODULE_NAME_LOG_FLAG__ = ATOMIC_FLAG_INIT; \\\n (!__MODULE_NAME_LOG_FLAG__.test_and_set(std::memory_order_relaxed)); \\\n })\n\n#define MODULE_NAME_EVERY_N(n) \\\n ({ \\\n MODULE_NAME_ASSERT((n) > 0 && \"should not be less than 1\"); \\\n static std::atomic<uint64_t> __MODULE_NAME_LOG_OCCURENCE_N__{(n)}; \\\n (__MODULE_NAME_LOG_OCCURENCE_N__.fetch_add(1, std::memory_order_relaxed) % (n) == 0); \\\n })\n\n#define MODULE_NAME_WITHIN_N(ms) \\\n ({ \\\n MODULE_NAME_ASSERT((ms) > 0 && \"should not be less than 1\"); \\\n bool __MODULE_NAME_LOG_WRITEABLE__ = false; \\\n static std::chrono::time_point<std::chrono::steady_clock> __MODULE_NAME_TIME_POINT__; \\\n static std::atomic_flag __MODULE_NAME_LOG_CONDITION_FLAG__ = ATOMIC_FLAG_INIT; \\\n if (!__MODULE_NAME_LOG_CONDITION_FLAG__.test_and_set(std::memory_order_acquire)) { \\\n auto __MODULE_NAME_NOWS__ = std::chrono::steady_clock::now(); \\\n int64_t __MODULE_NAME_DIFF_N__ = std::chrono::duration_cast<std::chrono::milliseconds>( \\\n __MODULE_NAME_NOWS__ - __MODULE_NAME_TIME_POINT__) \\\n .count(); \\\n if (__MODULE_NAME_DIFF_N__ > ms) { \\\n __MODULE_NAME_TIME_POINT__ = __MODULE_NAME_NOWS__; \\\n __MODULE_NAME_LOG_WRITEABLE__ = true; \\\n } \\\n __MODULE_NAME_LOG_CONDITION_FLAG__.clear(std::memory_order_release); \\\n } \\\n __MODULE_NAME_LOG_WRITEABLE__; \\\n })\n\n#define MODULE_NAME_INFO_ONCE(format, args...) MODULE_NAME_INFO_IF(MODULE_NAME_ONCE(), format, ##args)\n#define MODULE_NAME_WARN_ONCE(format, args...) MODULE_NAME_WARN_IF(MODULE_NAME_ONCE(), format, ##args)\n#define MODULE_NAME_ERROR_ONCE(format, args...) MODULE_NAME_ERROR_IF(MODULE_NAME_ONCE(), format, ##args)\n#define MODULE_NAME_CRITICAL_ONCE(format, args...) MODULE_NAME_CRITICAL_IF(MODULE_NAME_ONCE(), format, ##args)\n\n#define MODULE_NAME_INFO_EVERY_N(N, format, args...) MODULE_NAME_INFO_IF(MODULE_NAME_EVERY_N(N), format, ##args)\n#define MODULE_NAME_WARN_EVERY_N(N, format, args...) MODULE_NAME_WARN_IF(MODULE_NAME_EVERY_N(N), format, ##args)\n#define MODULE_NAME_ERROR_EVERY_N(N, format, args...) MODULE_NAME_ERROR_IF(MODULE_NAME_EVERY_N(N), format, ##args)\n#define MODULE_NAME_CRITICAL_EVERY_N(N, format, args...) \\\n MODULE_NAME_CRITICAL_IF(MODULE_NAME_EVERY_N(N), format, ##args)\n\n#define MODULE_NAME_INFO_EVERY_SECOND(format, args...) MODULE_NAME_INFO_IF(MODULE_NAME_WITHIN_N(1000), format, ##args)\n#define MODULE_NAME_WARN_EVERY_SECOND(format, args...) MODULE_NAME_WARN_IF(MODULE_NAME_WITHIN_N(1000), format, ##args)\n#define MODULE_NAME_ERROR_EVERY_SECOND(format, args...) \\\n MODULE_NAME_ERROR_IF(MODULE_NAME_WITHIN_N(1000), format, ##args)\n#define MODULE_NAME_CRITICAL_EVERY_SECOND(format, args...) \\\n MODULE_NAME_CRITICAL_IF(MODULE_NAME_WITHIN_N(1000), format, ##args)\nppt在这里 https://www.modb.pro/topic/640976
\n\n\n\n主要是提了个缓存加速
\n\n其他的就是常规的东西，raft和分布式文件系统相关优化(raft,存储纠删码，NWR，攒批写（小文件聚集写）等等)
\n\nWorkload分析和对应的挑战
\n\n
如何线性扩展
\n\n![\"\"](\"https://wanghenshui.github.io/assets/bytedance-tsdc.png\")
编码优化
\n\n观察数据特征
\n\nDatapoint Set
\n\n查询优化
\n\nKhronos存储引擎
\n\n![\"\"](\"https://wanghenshui.github.io/assets/bytedance-khronos1.png\")
Memtable
\n\nSST
\n\nMetric格式
\n\nFlush优化
\n\nSSTable查询优化
\n\n他们的工作做的确实挺多
\n\n就是回顾架构设计
\n\n主从
\n\n存算分离一写多读
\n\n阿里第二代serverless设计
\n\n其实阿里这么玩的前提是有一个资源池管理
\n", "date_published": "Fri, 19 Jul 2024 00:00:00 +0000" }, { "id": "https://wanghenshui.github.io/2024/07/19/2023-qcon-gz.html", "url": "https://wanghenshui.github.io/2024/07/19/2023-qcon-gz.html", "title": "qcon2023广州PPT速览", "content_html": "ppt在这里 https://www.modb.pro/topic/640977
\n\n只有两个感兴趣
\n\nhttps://wanghenshui.github.io/pdf/byconity.pdf
\n\nhttps://wanghenshui.github.io/pdf/antkv.pdf
\n\n\n\nWiscKey的rocksdb改造工作
\n\nAntKV 核心功能
\nScan优化
\n\nkv分离现状
\n优化策略，并发prefetch
\n![\"\"](\"https://wanghenshui.github.io/assets/antkv1.png\")
新的问题
\n原因
\n优化策略 Diffkv ATC 21: Differentiated Key-Value Storage Management for Balanced I/O Performance
\n\n核心思路
\n![\"\"](\"https://wanghenshui.github.io/assets/antkv2.png\")
![\"\"](\"https://wanghenshui.github.io/assets/antkv3.png\")
收益 写入降低30% 但scan提升巨大
\n\n这种还是要考虑业务来使用，但是这个工作是很亮眼的
\n\n借助 Learned Index 优化查询
\n\nLearned Index主要是要设计构建算法，这里需要展开一下
\n\n![\"\"](\"https://wanghenshui.github.io/assets/antkv-li1.png\")
![\"\"](\"https://wanghenshui.github.io/assets/antkv-li2.png\")
因为实际 SST 保存的 key 为 string 类型，非 integer，因此需要进行转换
\n![\"\"](\"https://wanghenshui.github.io/assets/antkv-li3.png\")
Learned Index非常小，读效率非常高
\n\nLearned Index: 生成过程
\n\nclickhouse痛点
\n架构
\n\n设计考虑
\n\n数据缓存
\n\nByConity事务
\n中⼼授时服务TSO(TimeStamp Oracle)
\n说的东西还是非常多的，直接看pdf比我复述直观
\n\nhttps://wanghenshui.github.io/pdf/byconity.pdf
\n", "date_published": "Fri, 19 Jul 2024 00:00:00 +0000" }, { "id": "https://wanghenshui.github.io/2024/07/15/rocksdb-invalidation-cache.html", "url": "https://wanghenshui.github.io/2024/07/15/rocksdb-invalidation-cache.html", "title": "rocksdb的cache过期问题讨论", "content_html": "如果blockcache被compaction搞失效了，有没有一种逻辑自动重填blockcache，降低cache miss？
\n\n毕竟在blockcache中属于热数据
\n\n另外ingest文件能不能同时预热 blockcache？
\n\n结论:上层加row cache绕过
\n\nrocksdb本身不好解决这个问题，太业务了
\n\n另外 rocksdb rowcache有兼容性问题(deleterange breaking ,v9)，不能用
\n\n\n\n维护一个compaction buffer https://nan01ab.github.io/2018/07/FD-tree-and-LSbM-tree.html
\n\n改进LSM Row Cache缓存热点行
\n\n减少Compaction的粒度；减少Compaction过程中改动的数据；Compaction过程中针对已有的缓存数据做定点更新（增量替换）
\n\n北京大学，vldb‘20 [18]，采用机器学习方法解决X-Engine中cache miss问题
\n\n在X-Engine实际运行中，由于后台异步数据合并任务造成的大面积缓存失效问题。之前也有论文提出这种问题，具体解决是多增一个buffer cache，空间换效率。
\n\nLeaper采用机器学习算法，预测一个 compaction 任务在执行过程中和刚刚结束执行时，数据库上层 SQL 负载可能会访问到数据记录并提前将其加载进 cache 中，从而实现降低 cache miss，提高 QPS 稳定性的目的
\n\nrocksdb本身没有这种信息输入接口，只能根据blockcache存在过这个信息来来加载
\n\n可以提供一个predict_op, 给上层判定是否回填
\n\n显然是业务驱动
\n\nblockcache改成cache key/cache adderss并且2Q策略维护
\n\n有个公式判定，较复杂 https://linqy71.github.io/2020/12/22/AC-Key/
\n\n实现难度也比较大
\n\n如果考虑单独做一个2Q cache放在业务上层，外部数据导入就扫一遍 cache上，实现更简单
\n\n我之前有个类似想法 FastCHD + 动态hashmap 定期重构
\n\n感觉内存不可控
\n\n另外cache的其他问题就不展开了，另一个话题
\n\nblockcache miss了，磁盘swap再撑一下，最后查FS/S3
\n", "date_published": "Mon, 15 Jul 2024 00:00:00 +0000" }, { "id": "https://wanghenshui.github.io/2024/07/07/user-group-001.html", "url": "https://wanghenshui.github.io/2024/07/07/user-group-001.html", "title": "这段代码能这段代码有办法做simd加速吗", "content_html": "群友讨论第一期, 这个栏目要多谢群友交流
\n\n本期鸣谢群友 @核聚变引擎 @mwish
群友核聚变引擎 提问
\n\n这段代码有办法做simd加速吗，感觉无解\nhttps://gcc.godbolt.org/z/GjjEfz1e8
\n\n代码如下
\n\n#include <cstdint>\n\nconst int M = (1 << 14);\nuint16_t bins[64] = {0};\nuint8_t data[M] = {0};\n\n// 统计 data 中 0~63 的频数，不考虑高 2 位\nvoid histgram(uint16_t *bins, uint8_t *data) {\n for (int i = 0; i < M; i++) {\n uint8_t x = data[i] & 0x3f;\n bins[x]++;\n }\n}\n\n这种计算是统计型，关键是统计M输入到64，不是多对1类型的计算，比如popcnt
\n\n优化角度无非是攒批/分片
\n\n使用chatgpt帮咱写了一个版本
\n\nvoid histgram_avx(uint16_t *bins, uint8_t *data) {\n __m256i mask = _mm256_set1_epi8(0x3F);\n for (int i = 0; i < M; i += 32) {\n __m256i chunk = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(data + i));\n chunk = _mm256_and_si256(chunk, mask);\n\n uint8_t tmp[32];\n _mm256_storeu_si256(reinterpret_cast<__m256i*>(tmp), chunk);\n\n for (int j = 0; j < 32; ++j) {\n bins[tmp[j]]++;\n }\n }\n}\n这种加速其实效果非常有限，M也不大，性能提升并不明显
\n\n群友mwish提供了一个分片的版本
\n\nusing Bucket = std::array<uint16_t, 64>;\n\n// 统计 data 中 0~63 的频数，不考虑高 2 位\nstd::array<Bucket, 8> histgram_mwish(uint16_t *bins, uint8_t *data) {\n std::array<Bucket, 8> temp_bins;\n for (int i = 0; i < M / 8; i++) {\n for (int idx = 0; idx < 8; ++idx) {\n uint8_t x = data[i * 8 + idx] % 64;\n temp_bins[idx][x]++;\n }\n }\n return temp_bins;\n}\n我压测了一版数据，机器是2019 mbp 性能一般，结果如下
\n\n2024-07-07T18:16:00+08:00\nRunning ./build/bm_histgram\nRun on (12 X 2600 MHz CPU s)\nCPU Caches:\n L1 Data 32 KiB\n L1 Instruction 32 KiB\n L2 Unified 256 KiB (x6)\n L3 Unified 12288 KiB\nLoad Average: 2.84, 3.07, 2.77\n--------------------------------------------------------------------------------\nBenchmark Time CPU Iterations\n--------------------------------------------------------------------------------\nBM_histgram_mwish/iterations:10000000 7065 ns 7019 ns 10000000\nBM_histgram_base/iterations:10000000 6836 ns 6769 ns 10000000\nBM_histgram_avx/iterations:10000000 7228 ns 7179 ns 10000000\n能看到性能差异基本可以无视，优化反而还慢了
\n\n\n\n有什么错误或者有什么改善的思路，欢迎提出
\n", "date_published": "Sun, 07 Jul 2024 00:00:00 +0000" }, { "id": "https://wanghenshui.github.io/2024/06/30/mm-latency.html", "url": "https://wanghenshui.github.io/2024/06/30/mm-latency.html", "title": "内存管理机制针对延迟的优化", "content_html": "原文
\n\nhttps://johnnysswlab.com/latency-sensitive-application-and-the-memory-subsystem-part-2-memory-management-mechanisms
\n\nhttps://www.jabperf.com/how-to-deter-or-disarm-tlb-shootdowns/
\n\n\n\n这篇文章的视角比较奇怪，可能和已知的信息不同，目标是低延迟避免内存机制影响
\n\npage fault会引入延迟，所以要破坏page fault的生成条件 怎么做？
\n\n尽可能分配好，而不是用到在分配，有概率触发page fault
\n\n这里展开讲讲，内容来自 这里
\n\ncat /proc/interrupts
\n\n观察tlb shutdown
\n\n CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 \n\n45: 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0\n46: 192342 0 0 0 0 0 0 0 PCI-MSI-edge ahci\n47: 14 0 0 0 0 0 0 0 PCI-MSI-edge mei\n\nNMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts\nLOC: 552219 1010298 2272333 3179890 1445484 1226202 1800191 1894825 Local timer interrupts\nSPU: 0 0 0 0 0 0 0 0 Spurious interrupts\n\nIWI: 0 0 0 0 0 0 0 0 IRQ work interrupts\nRTR: 7 0 0 0 0 0 0 0 APIC ICR read retries\nRES: 18708 9550 771 528 129 170 151 139 Rescheduling interrupts\nCAL: 711 934 1312 1261 1446 1411 1433 1432 Function call interrupts\nTLB: 4493 6108 73789 5014 1788 2327 1967 914 TLB shootdowns\n\n抓堆栈 native_flush_tlb_others很多 numa自平衡开了 关闭sysctl -w numa_balancing=0
TLB失效的原理 IPI触发
\n\n几个关键点
\n\n核心越多影响越大
\n\n怎么解决？
\n\n预先分配内存，然后从不释放内存。分配器优化/使用mallopt()和mlockall()
\n\n内核优化，不走IPI？
\n\n测试numa 代码 madvise
\n\npthread_barrier_t barrier;\n\nvoid* madv(void* mem)\n{\n pthread_barrier_wait(&barrier);\n\n std::chrono::steady_clock::time_point begin = std::chrono::steady_clock::now();\n for (int i = 0;i < 1'000'000; i++)\n {\n madvise(*((char**) mem), 4096, MADV_DONTNEED);\n }\n std::chrono::steady_clock::time_point end = std::chrono::steady_clock::now();\n std::cout << \"Time difference = \" << std::chrono::duration_cast<std::chrono::milliseconds>(end - begin).count() << \"[ms]\" << std::endl;\n return 0;\n}\n\nint main(int argc, char** argv)\n{\n void *mem;\n posix_memalign(&mem, 4096, 8192);\n\n auto thread_count = atoi(argv[1]);\n auto barrier_count = thread_count + 1;\n pthread_barrier_init(&barrier, nullptr, barrier_count);\n\n pthread_t threads[thread_count];\n pthread_attr_t attr;\n pthread_attr_init(&attr);\n pthread_attr_setscope(&attr, PTHREAD_SCOPE_SYSTEM);\n cpu_set_t cpus;\n\n for (int i = 0; i < thread_count; i++)\n {\n CPU_ZERO(&cpus);\n CPU_SET(i, &cpus);\n pthread_attr_setaffinity_np(&attr, sizeof(cpu_set_t), &cpus);\n pthread_create(&threads[i], &attr, madv, &mem);\n }\n\n sleep(2); pthread_barrier_wait(&barrier);\n\n for (int i = 0; i < thread_count; i++)\n {\n pthread_join(threads[i], nullptr);\n }\n return 0;\n}\nperf抓 TLB shootdown
\n\n[root@eltoro]# perf stat -r 3 -e probe:native_flush_tlb_others,tlb_flush.stlb_any chrt -f 90 ./madv 2\n\n Performance counter stats for 'chrt -f 90 ./madv 2' (3 runs):\n\n 3,958,266 probe:native_flush_tlb_others ( +- 0.28% )\n 7,928,801 tlb_flush.stlb_any ( +- 0.24% )\n \n[root@eltoro]# perf stat -r 3 -e probe:native_flush_tlb_others,tlb_flush.stlb_any chrt -f 90 ./madv 8\n\n Performance counter stats for 'chrt -f 90 ./madv 8' (3 runs):\n\n 15,946,430 probe:native_flush_tlb_others ( +- 0.03% )\n 123,288,313 tlb_flush.stlb_any ( +- 0.05% )\n \n[root@eltoro]# perf stat -r 3 -e probe:native_flush_tlb_others,tlb_flush.stlb_any chrt -f 90 ./madv 14\n\n Performance counter stats for 'chrt -f 90 ./madv 14' (3 runs):\n\n 27,986,605 probe:native_flush_tlb_others ( +- 0.01% )\n 376,502,522 tlb_flush.stlb_any ( +- 0.01% )\n抓一下调用频次
\n\n[root@eltoro]# funclatency.py -p 37344 native_flush_tlb_others\nTracing 1 functions for \"native_flush_tlb_others\"... Hit Ctrl-C to end.\n^C\n nsecs : count distribution\n 0 -> 1 : 0 | |\n 2 -> 3 : 0 | |\n 4 -> 7 : 0 | |\n 8 -> 15 : 0 | |\n 16 -> 31 : 0 | |\n 32 -> 63 : 0 | |\n 64 -> 127 : 0 | |\n 128 -> 255 : 0 | |\n 256 -> 511 : 0 | |\n 512 -> 1023 : 87926 | |\n 1024 -> 2047 : 3861380 |****************************************|\n 2048 -> 4095 : 32924 | |\n 4096 -> 8191 : 3628 | |\n 8192 -> 16383 : 543 | |\n 16384 -> 32767 : 526 | |\n 32768 -> 65535 : 7 | |\n 65536 -> 131071 : 0 | |\n 131072 -> 262143 : 0 | |\n 262144 -> 524287 : 0 | |\n 524288 -> 1048575 : 0 | |\n 1048576 -> 2097151 : 0 | |\n 2097152 -> 4194303 : 11 | |\n\navg = 1398 nsecs, total: 5582920090 nsecs, count: 3991212\n\n[root@eltoro]# funclatency.py -p 37600 native_flush_tlb_others\nTracing 1 functions for \"native_flush_tlb_others\"... Hit Ctrl-C to end.\n^C\n nsecs : count distribution\n 0 -> 1 : 0 | |\n 2 -> 3 : 0 | |\n 4 -> 7 : 0 | |\n 8 -> 15 : 0 | |\n 16 -> 31 : 0 | |\n 32 -> 63 : 0 | |\n 64 -> 127 : 0 | |\n 128 -> 255 : 0 | |\n 256 -> 511 : 0 | |\n 512 -> 1023 : 24 | |\n 1024 -> 2047 : 224605 | |\n 2048 -> 4095 : 9608508 |****************************************|\n 4096 -> 8191 : 6012919 |************************* |\n 8192 -> 16383 : 18881 | |\n 16384 -> 32767 : 2468 | |\n 32768 -> 65535 : 66 | |\n 65536 -> 131071 : 41 | |\n 131072 -> 262143 : 39 | |\n 262144 -> 524287 : 16 | |\n 524288 -> 1048575 : 72 | |\n 1048576 -> 2097151 : 0 | |\n 2097152 -> 4194303 : 86 | |\n 4194304 -> 8388607 : 3 | |\n\navg = 3891 nsecs, total: 62126896400 nsecs, count: 15963897\n\n[root@eltoro]# funclatency.py -p 37786 native_flush_tlb_others\nTracing 1 functions for \"native_flush_tlb_others\"... Hit Ctrl-C to end.\n^C\n nsecs : count distribution\n 0 -> 1 : 0 | |\n 2 -> 3 : 0 | |\n 4 -> 7 : 0 | |\n 8 -> 15 : 0 | |\n 16 -> 31 : 0 | |\n 32 -> 63 : 0 | |\n 64 -> 127 : 0 | |\n 128 -> 255 : 0 | |\n 256 -> 511 : 0 | |\n 512 -> 1023 : 23 | |\n 1024 -> 2047 : 158263 | |\n 2048 -> 4095 : 990737 |* |\n 4096 -> 8191 : 22814615 |****************************************|\n 8192 -> 16383 : 3699448 |****** |\n 16384 -> 32767 : 15401 | |\n 32768 -> 65535 : 676 | |\n 65536 -> 131071 : 53 | |\n 131072 -> 262143 : 83 | |\n 262144 -> 524287 : 28 | |\n 524288 -> 1048575 : 169 | |\n 1048576 -> 2097151 : 179 | |\n 2097152 -> 4194303 : 131 | |\n\navg = 6610 nsecs, total: 184948768204 nsecs, count: 27976588\n原文
\n\nhttps://johnnysswlab.com/latency-sensitive-applications-and-the-memory-subsystem-keeping-the-data-in-the-cache/
\n\n\n\nwhile循环，没干活，干活逻辑是数据访问，那没干活分支应该可以热数据
\n\n比如原来的逻辑
\n\ntd::unordered_map<int32_t, order> my_orders;\n...\npacket_t* p;\nwhile(!exit) {\n p = get_packet();\n // If packet arrived\n if (p) {\n // Check if the identifier is known to us\n auto it = my_orders.find(p->id);\n if (it != my_orders.end()) {\n send_answer(p->origin, it->second);\n }\n }\n}\nwhile里是个干活逻辑，但是有个大的if，我们可以把这个if拆出来分成干活不干活两个逻辑
\n\nstd::unordered_map<int32_t, order> my_orders;\n...\npacket_t* p;\nint64_t total_random_found = 0;\nwhile(!exit) {\n // 增加个检查header 然后再判断packet，不满足就去warm\n // 如果header没满足，packet必不满足\n if (packet_header_arrived()) {\n p = get_packet();\n // If packet arrived\n if (p) {\n // Check if the identifier is known to us\n auto it = my_orders.find(p->id);\n if (it != my_orders.end()) {\n send_answer(p->origin, it->second);\n }\n }\n } else {\n // 不干活就Cache warming \n auto random_id = get_random_id();\n auto it = my_orders.find(random_id);\n // 随便干点啥避免被编译器优化掉\n total_random_found += (it != my_orders.end());\n }\n}\nstd::cout << \"Total random found \" << total_random_found << \"\\n\";\n当然这种cache warm不一定非得随机，有可能副作用
\n\n可以从历史值来用，有个词怎么说来着，启发式
\n\n硬件层也有cache warm 比如 intel
\n\n其实就是prefetch clflush那套，如果你知道具体访问哪个，那prefetch确实是比较高效的
\n\namd也有 L3 Cache Range Reservation 不过没例子
\n\n作者测试了软件模拟cache warm，随机访问
\n\n数据，迭代多次的延迟，越小越好
\n\n| hashmap数据量 | \n正常访问hashmap | \n没有访问的时候只warm 0 | \n没有访问的时候随机warm | \n
|---|---|---|---|
| 1 K | \n226.1 (219.0) | \n213.3 (205.1) | \n132.5 (67.3) | \n
| 4 K | \n324.7 (296.3) | \n350.7 (331.3) | \n140.1 (95.4) | \n
| 16 K | \n396.8 (341.1) | \n389.1 (354.5) | \n208.7 (134.5) | \n
| 64 K | \n425.5 (376.1) | \n416.0 (360.6) | \n232.1 (152.6) | \n
| 256 K | \n514.2 (451.5) | \n473.3 (480.6) | \n338.8 (317.6) | \n
| 1 M | \n599.8 (550.2) | \n615.1 (573.6) | \n466.3 (429.8) | \n
| 4 M | \n702.1 (647.0) | \n619.7 (649.2) | \n531.3 (508.3) | \n
| 16 M | \n756.7 (677.6) | \n668.8 (707.4) | \n543.2 (499.9) | \n
| 64 M | \n769.1 (702.3) | \n735.9 (734.2) | \n641.0 (774.4) | \n
能看到随机访问 随机warm效果显著
\n\n和群友讨论
\n\nyangbowen认为没用，prefetch有用，手动warm当cpu傻逼(我感觉他没看懂这个例子)
\n\nmwish给了一些prefetch的资料 https://www.cs.cmu.edu/~chensm/papers/hashjoin_tods_preliminary.pdf
\n\n大家接触的多的例子就是prefetch，这种模拟cache warm还是比较少见的
\n", "date_published": "Sun, 30 Jun 2024 00:00:00 +0000" } ] }