活动介绍
file-type

MySQL Innodb_page_size性能对比:16K vs 8K

下载需积分: 0 | 415KB | 更新于2024-08-03 | 68 浏览量 | 1 下载量 举报 收藏
download 立即下载
"本文主要探讨了MySQL中InnoDB存储引擎的数据页大小对系统性能的影响,通过对innodb_page_size参数设置为16K和8K进行压力测试,对比分析了不同数据页大小下的CPU使用率和读写吞吐量。" 在MySQL数据库中,InnoDB存储引擎的数据页(也称为页或Block)是存储数据的基本单位。自MySQL 5.6版本起,引入了innodb_page_size参数,允许用户将数据页大小调整为8K、4K或默认的16K。这个参数必须在数据库初始化阶段就设定,一旦表已经创建,再更改此参数会导致启动错误。 本次性能测试是在一台配置为R710服务器上进行的,具有72GB内存和6块300GB 15000转硬盘组成的RAID10阵列,操作系统采用XFS文件系统。测试时配置了一系列InnoDB优化参数,如: 1. innodb_buffer_pool_size=48G - 设置InnoDB缓冲池大小为48GB,用于缓存数据页以提高I/O性能。 2. innodb_buffer_pool_instances=8 - 分割缓冲池为8个实例,提升并发性能。 3. innodb_flush_method=O_DIRECT - 使用O_DIRECT直接I/O模式,减少文件系统的缓存影响。 4. innodb_file_per_table=1 - 每个表单独存储为一个.ibd文件,便于管理。 5. innodb_read_io_threads=16 - 设置16个读取I/O线程,提高读取速度。 6. innodb_write_io_threads=16 - 设置16个写入I/O线程,加速写入操作。 7. innodb_io_capacity=2000 - 设定I/O子系统的最大能力为2000 IOPS。 8. innodb_log_files_in_group=3 - 设置日志文件组的数量。 9. innodb_flush_log_at_trx_commit=0 - 事务提交时不立即刷新日志缓冲区,优化性能。 10. innodb_log_file_size=1024M - 日志文件大小设置为1GB。 11. innodb_max_dirty_pages_pct=90 - 允许脏页占用缓冲池的最大比例,这里是90%。 测试工具使用了Sysbench,通过oltp测试场景模拟读写操作,创建名为sbtest的表,包含1亿条记录,总大小为24GB。测试参数包括读写混合操作,100个并发线程,最大请求次数为100万次。 测试结果显示: 1. 当innodb_page_size设置为16K时,CPU使用率较低,平均约20%,这表明16K的数据页大小对CPU负载有较好的控制。 2. 而当innodb_page_size设置为8K时,CPU使用率升高到30%~40%,虽然对CPU产生了更多压力,但在select吞吐量方面表现优于16K设置。 综合来看,选择8K还是16K的数据页大小取决于具体的应用场景。对于CPU资源有限且强调查询性能的系统,可能8K更合适;而如果CPU资源充足,更注重降低CPU负载,16K可能是更好的选择。在实际应用中,应根据业务需求和硬件资源状况,通过调整innodb_page_size参数来优化性能。

相关推荐

filetype

2025-06-19 13:54:33 10592 [Note] C:\Program Files\MySQL\MySQL Server 5.6\bin\mysqld.exe: Normal shutdown 2025-06-19 13:54:33 10592 [Note] Giving 15 client threads a chance to die gracefully 2025-06-19 13:54:33 10592 [Note] Event Scheduler: Killing the scheduler thread, thread id 116 2025-06-19 13:54:33 10592 [Note] Event Scheduler: Waiting for the scheduler thread to reply 2025-06-19 13:54:33 10592 [Note] Event Scheduler: Stopped 2025-06-19 13:54:33 10592 [Note] Event Scheduler: Purging the queue. 2 events 2025-06-19 13:54:33 10592 [Note] Shutting down slave threads 2025-06-19 13:54:35 10592 [Note] Forcefully disconnecting 3 remaining clients 2025-06-19 13:54:35 10592 [Warning] C:\Program Files\MySQL\MySQL Server 5.6\bin\mysqld.exe: Forcing close of thread 35 user: 'wcs' 2025-06-19 13:54:35 10592 [Warning] C:\Program Files\MySQL\MySQL Server 5.6\bin\mysqld.exe: Forcing close of thread 8 user: 'wcs' 2025-06-19 13:54:35 10592 [Warning] C:\Program Files\MySQL\MySQL Server 5.6\bin\mysqld.exe: Forcing close of thread 3729 user: 'wcs' 2025-06-19 13:54:35 10592 [Note] Binlog end 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'partition' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_FIELDS' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_INDEXES' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_SYS_TABLES' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_FT_CONFIG' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_FT_DELETED' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_METRICS' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_CMPMEM' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_CMP_RESET' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_CMP' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_LOCK_WAITS' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_LOCKS' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'INNODB_TRX' 2025-06-19 13:54:35 10592 [Note] Shutting down plugin 'InnoDB' 2025-06-19 13:54:35 10592 [Note] InnoDB: FTS optimize thread exiting. 2025-06-19 13:54:35 10592 [Note] InnoDB: Starting shutdown... 2025-06-19 13:54:37 10592 [Note] InnoDB: Shutdown completed; log sequence number 31950984072 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'BLACKHOLE' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'ARCHIVE' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'MRG_MYISAM' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'MyISAM' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'MEMORY' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'CSV' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'sha256_password' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'mysql_old_password' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'mysql_native_password' 2025-06-19 13:54:37 10592 [Note] Shutting down plugin 'binlog' 2025-06-19 13:54:37 10592 [Note] C:\Program Files\MySQL\MySQL Server 5.6\bin\mysqld.exe: Shutdown complete 2025-06-19 13:54:38 3896 [Note] Plugin 'FEDERATED' is disabled. 2025-06-19 13:54:38 289c InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator. 2025-06-19 13:54:38 3896 [Note] InnoDB: Using atomics to ref count buffer pool pages 2025-06-19 13:54:38 3896 [Note] InnoDB: The InnoDB memory heap is disabled 2025-06-19 13:54:38 3896 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions 2025-06-19 13:54:38 3896 [Note] InnoDB: Memory barrier is not used 2025-06-19 13:54:38 3896 [Note] InnoDB: Compressed tables use zlib 1.2.3 2025-06-19 13:54:38 3896 [Note] InnoDB: Not using CPU crc32 instructions 2025-06-19 13:54:38 3896 [Note] InnoDB: Initializing buffer pool, size = 8.0G 2025-06-19 13:54:38 3896 [Note] InnoDB: Completed initialization of buffer pool 2025-06-19 13:54:38 3896 [Note] InnoDB: Highest supported file format is Barracuda. 2025-06-19 13:54:39 3896 [Note] InnoDB: 128 rollback segment(s) are active. 2025-06-19 13:54:39 3896 [Note] InnoDB: Waiting for purge to start 2025-06-19 13:54:39 3896 [Note] InnoDB: 5.6.21 started; log sequence number 31950984072 2025-06-19 13:54:39 3896 [Note] Server hostname (bind-address): '*'; port: 3306 2025-06-19 13:54:39 3896 [Note] IPv6 is available. 2025-06-19 13:54:39 3896 [Note] - '::' resolves to '::'; 2025-06-19 13:54:39 3896 [Note] Server socket created on IP: '::'. 2025-06-19 13:54:39 3896 [Warning] InnoDB: Cannot open table mysql/slave_master_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2025-06-19 13:54:39 3896 [Warning] Info table is not ready to be used. Table 'mysql.slave_master_info' cannot be opened. 2025-06-19 13:54:39 3896 [Warning] InnoDB: Cannot open table mysql/slave_worker_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2025-06-19 13:54:39 3896 [Warning] InnoDB: Cannot open table mysql/slave_relay_log_info from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem. 2025-06-19 13:54:39 3896 [Warning] Info table is not ready to be used. Table 'mysql.slave_relay_log_info' cannot be opened. 2025-06-19 13:54:39 3896 [Note] Event Scheduler: Loaded 2 events 2025-06-19 13:54:39 3896 [Note] C:\Program Files\MySQL\MySQL Server 5.6\bin\mysqld.exe: ready for connections. Version: '5.6.21-log' socket: '' port: 3306 MySQL Community Server (GPL) 2025-06-19 13:54:39 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.22'. 2025-06-19 13:54:39 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.22'. 2025-06-19 13:54:39 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:39 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:39 3896 [Note] - fe80::659c:c088:9d63:4346%17 2025-06-19 13:54:39 3896 [Note] - fe80::659c:c088:9d63:4346%17 2025-06-19 13:54:39 3896 [Note] - 11.1.85.24 2025-06-19 13:54:39 3896 [Note] - 169.254.119.0 2025-06-19 13:54:39 26c InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:39 26c2025-06-19 13:54:39 2a90 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:39 2a90 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."feedercameradata" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."feedercameradata" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:39 2cb0 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:39 2cb0 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."alreadysortinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:39 2028 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:39 20282025-06-19 13:54:39 249c InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:39 249c InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."alreadysortinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."alreadysortinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:39 1c84 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:39 1c84 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."errorrecord" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:39 808 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:39 808 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."systemlog" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:41 2d04 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:41 2d04 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."devicestate_monitorinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:41 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.31'. 2025-06-19 13:54:41 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:41 3896 [Note] - fe80::e985:a0f1:2751:eada%17 2025-06-19 13:54:41 3896 [Note] - 169.254.170.71 2025-06-19 13:54:42 24b8 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:42 24b8 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."userinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:42 3896 [Warning] Hostname 'DESKTOP-1LL2I5F' does not resolve to '11.1.85.33'. 2025-06-19 13:54:42 3896 [Note] Hostname 'DESKTOP-1LL2I5F' has the following IP addresses: 2025-06-19 13:54:42 3896 [Note] - fe80::7082:fd95:7ec0:bcc8%17 2025-06-19 13:54:42 3896 [Note] - 169.254.255.9 2025-06-19 13:54:42 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.26'. 2025-06-19 13:54:42 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:42 3896 [Note] - fe80::daa7:903b:6a3c:c735%17 2025-06-19 13:54:42 3896 [Note] - 11.1.85.31 2025-06-19 13:54:42 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.22'. 2025-06-19 13:54:42 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:42 3896 [Note] - fe80::659c:c088:9d63:4346%17 2025-06-19 13:54:42 3896 [Note] - 11.1.85.37 2025-06-19 13:54:43 808 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:43 808 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."portcode_timing" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:43 25a0 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:43 25a0 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."helpmanual" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:45 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.30'. 2025-06-19 13:54:45 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:45 3896 [Note] - fe80::6b84:3a08:e1d0:e750%17 2025-06-19 13:54:45 3896 [Note] - 11.1.85.31 2025-06-19 13:54:45 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.24'. 2025-06-19 13:54:45 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:45 3896 [Note] - fe80::daa7:903b:6a3c:c735%17 2025-06-19 13:54:45 3896 [Note] - 11.1.85.28 2025-06-19 13:54:47 3896 [Warning] Hostname 'DESKTOP-S1Q82IV' does not resolve to '11.1.85.27'. 2025-06-19 13:54:47 3896 [Note] Hostname 'DESKTOP-S1Q82IV' has the following IP addresses: 2025-06-19 13:54:47 3896 [Note] - fe80::3711:88ce:2316:8596%17 2025-06-19 13:54:47 3896 [Note] - 169.254.226.201 2025-06-19 13:54:47 808 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2c6c InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2c6c2025-06-19 13:54:47 808 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."feedinginfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."feedinginfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 24b8 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 24b82025-06-19 13:54:47 78c InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."feedinginfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 78c InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."feedinginfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."alreadysortinfohist" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."arrival" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."arrival_copy" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."bagcheckalarmforsto" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."bagcode_base64" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."baglifeinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."bagscanresultinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."basisdata" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."billlog" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."billloghist" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."checkscaleinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."countinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."customerinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."departure" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."departure_copy" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."devicestateinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."devicestateinfooferrorclosewcs" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."eachscanresult" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."electricenergyrecord" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."event_log" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."fail_upload" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."failsortinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."failsortinfohist" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."failuploaddata" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."feedinginfo1" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."feedinginfohist" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."interfaceusagedetail" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."jdv2picinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."loading_occupancy" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."manualcodingresult" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."manualcodingresulthist" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."moduleversion" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."nccountinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."orderauto" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."packagecode" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."packageprintinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."port_bagid" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."port_printer" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."portandboxcode" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."portcode_timinghist" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."queue_task" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."readtarget" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."reuploadopccodedata" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."routingtable" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."scannersortinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."scanresult" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."show_billcode" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."sortcodeinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."statisticsofpackagesupplyrecords" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."switchshiftofsto" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."ts" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."upload_fail" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."uploaddata" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."uploadpicinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."user" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."useractioninfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."userrank_rankname" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."userrankinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."webpointparameter" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."webpointparameter2" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."yzgjsortinfo" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:47 2738 InnoDB: Error: Table "mysql"."innodb_table_stats" not found. 2025-06-19 13:54:47 2738 InnoDB: Error: Fetch of persistent statistics requested for table "db_wcs"."yzshift" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2025-06-19 13:54:49 3896 [Warning] Hostname 'DESKTOP-S1Q82IV' does not resolve to '11.1.85.25'. 2025-06-19 13:54:49 3896 [Note] Hostname 'DESKTOP-S1Q82IV' has the following IP addresses: 2025-06-19 13:54:49 3896 [Note] - fe80::a18:a682:b681:52a8%17 2025-06-19 13:54:49 3896 [Note] - 169.254.226.201 2025-06-19 13:54:49 3896 [Warning] Hostname 'DESKTOP-S1Q82IV' does not resolve to '11.1.85.39'. 2025-06-19 13:54:49 3896 [Note] Hostname 'DESKTOP-S1Q82IV' has the following IP addresses: 2025-06-19 13:54:49 3896 [Note] - fe80::416:a93d:15ac:92bc%17 2025-06-19 13:54:49 3896 [Note] - 169.254.226.201 2025-06-19 13:54:49 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.40'. 2025-06-19 13:54:49 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:49 3896 [Note] - fe80::e985:a0f1:2751:eada%17 2025-06-19 13:54:49 3896 [Note] - 11.1.85.28 2025-06-19 13:54:50 3896 [Warning] Hostname 'DESKTOP-S1Q82IV' does not resolve to '11.1.85.32'. 2025-06-19 13:54:50 3896 [Note] Hostname 'DESKTOP-S1Q82IV' has the following IP addresses: 2025-06-19 13:54:50 3896 [Note] - fe80::3711:88ce:2316:8596%17 2025-06-19 13:54:50 3896 [Note] - 169.254.226.201 2025-06-19 13:54:50 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.38'. 2025-06-19 13:54:50 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:50 3896 [Note] - fe80::e985:a0f1:2751:eada%17 2025-06-19 13:54:50 3896 [Note] - 169.254.178.147 2025-06-19 13:54:51 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.36'. 2025-06-19 13:54:51 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:51 3896 [Note] - fe80::e985:a0f1:2751:eada%17 2025-06-19 13:54:51 3896 [Note] - 169.254.170.71 2025-06-19 13:54:52 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.28'. 2025-06-19 13:54:52 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:52 3896 [Note] - fe80::daa7:903b:6a3c:c735%17 2025-06-19 13:54:52 3896 [Note] - 11.1.85.36 2025-06-19 13:54:53 3896 [Warning] Hostname 'DESKTOP-56G0R2L' does not resolve to '11.1.85.37'. 2025-06-19 13:54:53 3896 [Note] Hostname 'DESKTOP-56G0R2L' has the following IP addresses: 2025-06-19 13:54:53 3896 [Note] - fe80::daa7:903b:6a3c:c735%17 2025-06-19 13:54:53 3896 [Note] - 11.1.85.28 windows2019下安装的mysql5.6 今天启动报错1067,重启服务器后正常了,以上是收集的mysql日志,帮忙分析报错1067的原因

filetype
filetype
filetype

sudo cat /usr/local/mysql/data/VM-16-9-centos.err 2025-03-20T02:00:15.739959Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2025-03-20T02:00:15.740032Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled 2025-03-20T02:00:15.740057Z 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.7.36) starting as process 813856 ... 2025-03-20T02:00:15.747664Z 0 [Note] InnoDB: PUNCH HOLE support available 2025-03-20T02:00:15.747684Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2025-03-20T02:00:15.747688Z 0 [Note] InnoDB: Uses event mutexes 2025-03-20T02:00:15.747691Z 0 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier 2025-03-20T02:00:15.747695Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2025-03-20T02:00:15.747698Z 0 [Note] InnoDB: Using Linux native AIO 2025-03-20T02:00:15.747921Z 0 [Note] InnoDB: Number of pools: 1 2025-03-20T02:00:15.748013Z 0 [Note] InnoDB: Using CPU crc32 instructions 2025-03-20T02:00:15.749664Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2025-03-20T02:00:15.758255Z 0 [Note] InnoDB: Completed initialization of buffer pool 2025-03-20T02:00:15.760782Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2025-03-20T02:00:15.772580Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 2025-03-20T02:00:15.783164Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2025-03-20T02:00:15.783219Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2025-03-20T02:00:15.815854Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2025-03-20T02:00:15.816646Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.

filetype

### MySQL将小表作为驱动表的IO角度深度解析 在MySQL中,将数据量小的表作为驱动表(即JOIN操作中`FROM`后的第一个表)能显著提升性能,**核心原因在于减少磁盘I/O操作次数**。以下是具体机制分析: --- ### 一、JOIN操作的基本I/O流程 MySQL执行JOIN时采用 **Nested-Loop Join(NLJ)算法**,其I/O过程分为两步: 1. **读取驱动表数据**:将驱动表数据加载到内存 2. **匹配被驱动表数据**:逐行用驱动表数据扫描被驱动表 ```mermaid graph LR A[驱动表] -->|逐行读取| B[内存] B -->|用每行数据| C[扫描被驱动表] C -->|磁盘I/O| D[返回匹配行] ``` --- ### 二、小表驱动的I/O优势原理 #### 场景1:被驱动表有索引(Index Nested-Loop Join) 假设: - 驱动表 $A$:$M$ 行,占 $P_A$ 个磁盘页 - 被驱动表 $B$:$N$ 行,占 $P_B$ 个磁盘页($P_B \gg P_A$) - $B$ 表连接字段有索引(B+树高度 $H$) | 驱动表选择 | I/O 次数公式 | 关键说明 | |----------------|----------------------------------|----------| | **小表A驱动** | $P_A + M \times (H + 1)$ | 1. 加载 $A$ 表:$P_A$ 次 I/O
2. 每行 $A$ 数据通过索引查 $B$:$H$(索引访问)+1(数据页访问) | | **大表B驱动** | $P_B + N \times (H + 1)$ | 1. 加载 $B$ 表:$P_B$ 次 I/O(代价更高)
2. 每行 $B$ 数据查 $A$:索引高度 $H$ 相同 | **优势体现**: 当 $M \ll N$ 且 $P_A \ll P_B$ 时(小表驱动): - **磁盘扫描量减少**:$P_A$ 远小于 $P_B$(如 10页 vs 1000页) - **索引查找次数降低**:$M \times (H+1) \ll N \times (H+1)$ - **缓存利用率高**:小表更易完整载入内存(Buffer Pool),避免反复磁盘I/O > 📌 **案例**:$M=1000, N=1e6, H=3, P_A=10, P_B=1000$ > - 小表驱动:$10 + 1000 \times 4 = 4,010$ 次I/O > - 大表驱动:$1000 + 10^6 \times 4 = 4,001,000$ 次I/O > **性能差距约1000倍!** #### 场景2:被驱动表无索引(Block Nested-Loop Join) 当被驱动表无索引时,MySQL使用 **Block Nested-Loop Join(BNL)**: 1. 将驱动表分块加载到 `join_buffer`(大小由 `join_buffer_size` 决定) 2. 全表扫描被驱动表与内存中的块匹配 **I/O 成本公式**: $$ \text{Total I/O} = P_A + \left \lceil \frac{P_A \times \text{Page\_Size}}{\text{join\_buffer\_size}} \right \rceil \times P_B $$ **小表驱动的优势**: - 若 $P_A$ 足够小($\text{join\_buffer\_size} \geq P_A \times \text{Page\_Size}$): - 驱动表可一次性加载到内存 - **只需1次全表扫描 $B$ 表**:$Total I/O = P_A + P_B$ - 若大表驱动: - 需分多次加载($\left \lceil \frac{P_B \times \text{Page\_Size}}{\text{join\_buffer\_size}} \right \rceil$ 增大) - **多次全表扫描 $A$ 表**:$Total I/O = P_B + K \times P_A$($K$ 为分块数) > ⚠️ **极端案例**:$P_A=100, P_B=1000, \text{join\_buffer\_size}=1\text{MB}, \text{Page\_Size}=16\text{KB}$ > - 小表驱动:分块数 $K= \lceil (100×16)/1024 \rceil=2$ → $100 + 2×1000=2,100$ 次I/O > - 大表驱动:$K= \lceil (1000×16)/1024 \rceil=16$ → $1000 + 16×100=2,600$ 次I/O > 差距:**500次额外I/O**(约8MB磁盘读取) --- ### 三、其他I/O相关优化效应 1. **减少锁竞争**: - 小表扫描时间短,锁定资源更快释放 - 大表作为被驱动表时,扫描过程可触发**MRR优化**(Multi-Range Read),将随机I/O转为顺序I/O 2. **预热缓存效应**: - 频繁访问的小表常驻Buffer Pool - 后续JOIN操作直接命中内存,避免磁盘I/O 3. **顺序访问优势**: - 小表驱动时,被驱动表的索引扫描更可能触发**Batched Key Access(BKA)** - 将随机I/O合并为顺序I/O(尤其对SSD重要) --- ### 四、优化器如何选择驱动表? MySQL优化器通过成本模型自动选择: ```sql EXPLAIN SELECT ... FROM table_A JOIN table_B ...; ``` 优化器优先考虑: 1. **表大小**(通过 `SHOW TABLE STATUS` 估算) 2. **可用索引** 3. **连接缓冲区大小** 4. **I/O成本权重**(通常占总体成本70%+) > 💡 可通过 `STRAIGHT_JOIN` 强制驱动表顺序: > ```sql > SELECT ... FROM small_table STRAIGHT_JOIN large_table ... > ``` --- ### 结论:为什么小表驱动优化I/O? | 关键机制 | 效果 | |------------------------|--------------------------| | **减少驱动表加载I/O** | 小表占磁盘页少($P_A↓$) | | **降低匹配过程I/O** | 索引查找次数少($M↓$) | | **避免多次扫描大表** | BNL算法分块数减少 | | **提高缓存利用率** | 小表常驻内存 | **终极建议**: 1. **小表作为驱动表** 2. **被驱动表的连接字段必须建索引** 3. 适当增大 `join_buffer_size`(默认256KB) 4. 定期执行 `ANALYZE TABLE` 更新统计信息 --- ### 附录:I/O成本计算工具 ```sql -- 查看表数据页数量 SELECT table_name, ROUND((data_length + index_length) / 1024 / 16) AS pages -- InnoDB默认16KB/页 FROM information_schema.TABLES WHERE table_schema = 'your_db'; ``` 能不能详细解释一下这个sql

filetype

[root@node ~]# start-dfs.sh Starting namenodes on [node] Last login: 二 7月 8 16:00:18 CST 2025 from 192.168.1.92 on pts/0 Starting datanodes Last login: 二 7月 8 16:00:38 CST 2025 on pts/0 Starting secondary namenodes [node] Last login: 二 7月 8 16:00:41 CST 2025 on pts/0 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# start-yarn.sh Starting resourcemanager Last login: 二 7月 8 16:00:45 CST 2025 on pts/0 Starting nodemanagers Last login: 二 7月 8 16:00:51 CST 2025 on pts/0 [root@node ~]# mapred --daemon start historyserver [root@node ~]# jps 3541 ResourceManager 4007 Jps 2984 NameNode 3944 JobHistoryServer 3274 SecondaryNameNode [root@node ~]# mkdir -p /weblog [root@node ~]# cat > /weblog/access.log << EOF > 192.168.1.1,2023-06-01 10:30:22,/index.html > 192.168.1.2,2023-06-01 10:31:15,/product.html > 192.168.1.1,2023-06-01 10:32:45,/cart.html > 192.168.1.3,2023-06-01 11:45:30,/checkout.html > 192.168.1.4,2023-06-01 12:10:05,/index.html > 192.168.1.2,2023-06-01 14:20:18,/product.htm > EOF [root@node ~]# ls /weblog access.log [root@node ~]# hdfs dfs -mkdir -p /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# hdfs dfs -put /weblog/access.log /weblog/raw/ SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# hdfs dfs -ls /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 1 items -rw-r--r-- 3 root supergroup 269 2025-07-08 16:03 /weblog/raw/access.log [root@node ~]# cd /weblog [root@node weblog]# mkdir weblog-mapreduce [root@node weblog]# cd weblog-mapreduce [root@node weblog-mapreduce]# touch CleanMapper.java [root@node weblog-mapreduce]# vim CleanMapper.java import java.io.IOException; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; public class CleanMapper extends Mapper<LongWritable, Text, Text, NullWritable> { public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String[] fields = line.split(","); if(fields.length == 3) { String ip = fields[0]; String time = fields[1]; String page = fields[2]; if(ip.matches("\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}")) { String outputLine = ip + "," + time + "," + page; context.write(new Text(outputLine), NullWritable.get()); } } } } [root@node weblog-mapreduce]# touch CleanReducer.java [root@node weblog-mapreduce]# vim CleanReducer.java import java.io.IOException; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; public class CleanReducer extends Reducer<Text, NullWritable, Text, NullWritable> { public void reduce(Text key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException { context.write(key, NullWritable.get()); } } [root@node weblog-mapreduce]# touch LogCleanDriver.java [root@node weblog-mapreduce]# vim LogCleanDriver.java import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class LogCleanDriver { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "Web Log Cleaner"); job.setJarByClass(LogCleanDriver.class); job.setMapperClass(CleanMapper.class); job.setReducerClass(CleanReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# javac -classpath $(hadoop classpath) -d . *.java [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.class CleanReducer.class LogCleanDriver.class CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# jar cf logclean.jar *.class [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.class CleanReducer.class LogCleanDriver.class logclean.jar CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 1 items -rw-r--r-- 3 root supergroup 269 2025-07-08 16:03 /weblog/raw/access.log [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. ls: `/weblog/output': No such file or directory [root@node weblog-mapreduce]# hadoop jar logclean.jar LogCleanDriver /weblog/raw /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node weblog-mapreduce]# [root@node weblog-mapreduce]# mapred job -status job_1751961655287_0001 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Job: job_1751961655287_0001 Job File: hdfs://node:9000/tmp/hadoop-yarn/staging/history/done/2025/07/08/000000/job_1751961655287_0001_conf.xml Job Tracking URL : http://node:19888/jobhistory/job/job_1751961655287_0001 Uber job : false Number of maps: 1 Number of reduces: 1 map() completion: 1.0 reduce() completion: 1.0 Job state: SUCCEEDED retired: false reason for failure: Counters: 54 File System Counters FILE: Number of bytes read=287 FILE: Number of bytes written=552699 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=372 HDFS: Number of bytes written=269 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=1848 Total time spent by all reduces in occupied slots (ms)=2016 Total time spent by all map tasks (ms)=1848 Total time spent by all reduce tasks (ms)=2016 Total vcore-milliseconds taken by all map tasks=1848 Total vcore-milliseconds taken by all reduce tasks=2016 Total megabyte-milliseconds taken by all map tasks=1892352 Total megabyte-milliseconds taken by all reduce tasks=2064384 Map-Reduce Framework Map input records=6 Map output records=6 Map output bytes=269 Map output materialized bytes=287 Input split bytes=103 Combine input records=0 Combine output records=0 Reduce input groups=6 Reduce shuffle bytes=287 Reduce input records=6 Reduce output records=6 Spilled Records=12 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=95 CPU time spent (ms)=1050 Physical memory (bytes) snapshot=500764672 Virtual memory (bytes) snapshot=5614292992 Total committed heap usage (bytes)=379584512 Peak Map Physical memory (bytes)=293011456 Peak Map Virtual memory (bytes)=2803433472 Peak Reduce Physical memory (bytes)=207753216 Peak Reduce Virtual memory (bytes)=2810859520 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=269 File Output Format Counters Bytes Written=269 [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 2 items -rw-r--r-- 3 root supergroup 0 2025-07-08 16:34 /weblog/output/_SUCCESS -rw-r--r-- 3 root supergroup 269 2025-07-08 16:34 /weblog/output/part-r-00000 [root@node weblog-mapreduce]# hdfs dfs -cat /weblog/output/part-r-00000 | head -5 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 192.168.1.1,2023-06-01 10:30:22,/index.html 192.168.1.1,2023-06-01 10:32:45,/cart.html 192.168.1.2,2023-06-01 10:31:15,/product.html 192.168.1.2,2023-06-01 14:20:18,/product.htm 192.168.1.3,2023-06-01 11:45:30,/checkout.html [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = 5199f37c-a381-428a-be1b-0a2afaab8583 Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = f38c99b3-ff7c-4f61-ae07-6b21d86d7160 hive> CREATE EXTERNAL TABLE weblog ( > ip STRING, > access_time TIMESTAMP, > page STRING > ) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY ',' > LOCATION '/weblog/output'; OK Time taken: 1.274 seconds hive> select * from weblog; OK 192.168.1.1 2023-06-01 10:30:22 /index.html 192.168.1.1 2023-06-01 10:32:45 /cart.html 192.168.1.2 2023-06-01 10:31:15 /product.html 192.168.1.2 2023-06-01 14:20:18 /product.htm 192.168.1.3 2023-06-01 11:45:30 /checkout.html 192.168.1.4 2023-06-01 12:10:05 /index.html Time taken: 1.947 seconds, Fetched: 6 row(s) hive> select * from weblog limit 5; OK 192.168.1.1 2023-06-01 10:30:22 /index.html 192.168.1.1 2023-06-01 10:32:45 /cart.html 192.168.1.2 2023-06-01 10:31:15 /product.html 192.168.1.2 2023-06-01 14:20:18 /product.htm 192.168.1.3 2023-06-01 11:45:30 /checkout.html Time taken: 0.148 seconds, Fetched: 5 row(s) hive> hive> CREATE TABLE page_visits AS > SELECT > page, > COUNT(*) AS visits > FROM weblog > GROUP BY page > ORDER BY visits DESC; Query ID = root_20250708183002_ec44d1b4-af24-403c-bb67-380dfb6961c3 Total jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0002, Tracking URL = http://node:8088/proxy/application_1751961655287_0002/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0002 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-07-08 18:30:12,692 Stage-1 map = 0%, reduce = 0% 2025-07-08 18:30:16,978 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.8 sec 2025-07-08 18:30:23,184 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.66 sec MapReduce Total cumulative CPU time: 3 seconds 660 msec Ended Job = job_1751961655287_0002 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0003, Tracking URL = http://node:8088/proxy/application_1751961655287_0003/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0003 Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1 2025-07-08 18:30:35,969 Stage-2 map = 0%, reduce = 0% 2025-07-08 18:30:41,155 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.23 sec 2025-07-08 18:30:46,313 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.95 sec MapReduce Total cumulative CPU time: 2 seconds 950 msec Ended Job = job_1751961655287_0003 Moving data to directory hdfs://node:9000/hive/warehouse/page_visits MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.66 sec HDFS Read: 12379 HDFS Write: 251 SUCCESS Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 2.95 sec HDFS Read: 7308 HDFS Write: 150 SUCCESS Total MapReduce CPU Time Spent: 6 seconds 610 msec OK Time taken: 46.853 seconds hive> hive> describe page_visits; OK page string visits bigint Time taken: 0.214 seconds, Fetched: 2 row(s) hive> CREATE TABLE ip_visits AS > SELECT > ip, > COUNT(*) AS visits > FROM weblog > GROUP BY ip > ORDER BY visits DESC; Query ID = root_20250708183554_da402d08-af34-46f9-a33a-3f66ddd1a580 Total jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0004, Tracking URL = http://node:8088/proxy/application_1751961655287_0004/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0004 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-07-08 18:36:04,037 Stage-1 map = 0%, reduce = 0% 2025-07-08 18:36:09,250 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec 2025-07-08 18:36:14,393 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.3 sec MapReduce Total cumulative CPU time: 3 seconds 300 msec Ended Job = job_1751961655287_0004 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0005, Tracking URL = http://node:8088/proxy/application_1751961655287_0005/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0005 Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1 2025-07-08 18:36:27,073 Stage-2 map = 0%, reduce = 0% 2025-07-08 18:36:31,215 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.25 sec 2025-07-08 18:36:36,853 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.27 sec MapReduce Total cumulative CPU time: 3 seconds 270 msec Ended Job = job_1751961655287_0005 Moving data to directory hdfs://node:9000/hive/warehouse/ip_visits MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.3 sec HDFS Read: 12445 HDFS Write: 216 SUCCESS Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 3.27 sec HDFS Read: 7261 HDFS Write: 129 SUCCESS Total MapReduce CPU Time Spent: 6 seconds 570 msec OK Time taken: 44.523 seconds hive> [root@node weblog-mapreduce]# hive> [root@node weblog-mapreduce]# describe ip_visite; bash: describe: command not found... [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = 57dafc2a-afe2-41a4-8159-00f8d44b5add Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive Session ID = f866eae4-4cb4-4403-b7a2-7a52701c5a74 Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. hive> describe ip_visite; FAILED: SemanticException [Error 10001]: Table not found ip_visite hive> describe ip_visits; OK ip string visits bigint Time taken: 0.464 seconds, Fetched: 2 row(s) hive> SELECT * FROM page_visits; OK /index.html 2 /product.html 1 /product.htm 1 /checkout.html 1 /cart.html 1 Time taken: 2.095 seconds, Fetched: 5 row(s) hive> SELECT * FROM ip_visits; OK 192.168.1.2 2 192.168.1.1 2 192.168.1.4 1 192.168.1.3 1 Time taken: 0.176 seconds, Fetched: 4 row(s) hive> hive> [root@node weblog-mapreduce]# [root@node weblog-mapreduce]# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 48 Server version: 8.0.42 MySQL Community Server - GPL Copyright (c) 2000, 2025, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> CREATE DATABASE IF NOT EXISTS weblog_db; Query OK, 1 row affected (0.06 sec) mysql> USE weblog_db; Database changed mysql> CREATE TABLE IF NOT EXISTS page_visits ( -> page VARCHAR(255), -> visits BIGINT -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8; Query OK, 0 rows affected, 1 warning (0.05 sec) mysql> SHOW TABLES; +---------------------+ | Tables_in_weblog_db | +---------------------+ | page_visits | +---------------------+ 1 row in set (0.00 sec) mysql> DESCRIBE page_visits; +--------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------+--------------+------+-----+---------+-------+ | page | varchar(255) | YES | | NULL | | | visits | bigint | YES | | NULL | | +--------+--------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) mysql> CREATE TABLE IF NOT EXISTS ip_visits ( -> ip VARCHAR(15), -> visits BIGINT -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8; Query OK, 0 rows affected, 1 warning (0.02 sec) mysql> SHOW TABLES; +---------------------+ | Tables_in_weblog_db | +---------------------+ | ip_visits | | page_visits | +---------------------+ 2 rows in set (0.01 sec) mysql> DESC ip_visits; +--------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------+-------------+------+-----+---------+-------+ | ip | varchar(15) | YES | | NULL | | | visits | bigint | YES | | NULL | | +--------+-------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) mysql> ^C mysql> [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = f34e6971-71ae-4aa5-aa22-895061f33bdf Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = f7a06e76-e117-4fbb-9ee8-09fdfd002104 hive> DESCRIBE FORMATTED page_visits; OK # col_name data_type comment page string visits bigint # Detailed Table Information Database: default OwnerType: USER Owner: root CreateTime: Tue Jul 08 18:30:47 CST 2025 LastAccessTime: UNKNOWN Retention: 0 Location: hdfs://node:9000/hive/warehouse/page_visits Table Type: MANAGED_TABLE Table Parameters: COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} bucketing_version 2 numFiles 1 numRows 5 rawDataSize 70 totalSize 75 transient_lastDdlTime 1751970648 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat: org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: serialization.format 1 Time taken: 1.043 seconds, Fetched: 32 row(s) hive> 到这里就不会了 6.2.2sqoop导出格式 6.2.3导出page_visits表 6.2.4导出到ip_visits表 6.3验证导出数据 6.3.1登录MySQL 6.3.2执行查询

filetype
filetype

2025-03-10T06:43:37.165038Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2025-03-10T06:43:37.165092Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled 2025-03-10T06:43:37.165104Z 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.7.18) starting as process 26836 ... 2025-03-10T06:43:37.169695Z 0 [Note] InnoDB: PUNCH HOLE support not available 2025-03-10T06:43:37.169704Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2025-03-10T06:43:37.169706Z 0 [Note] InnoDB: Uses event mutexes 2025-03-10T06:43:37.169708Z 0 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier 2025-03-10T06:43:37.169709Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3 2025-03-10T06:43:37.169711Z 0 [Note] InnoDB: Using Linux native AIO 2025-03-10T06:43:37.169831Z 0 [Note] InnoDB: Number of pools: 1 2025-03-10T06:43:37.169878Z 0 [Note] InnoDB: Using CPU crc32 instructions 2025-03-10T06:43:37.170938Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2025-03-10T06:43:37.178293Z 0 [Note] InnoDB: Completed initialization of buffer pool 2025-03-10T06:43:37.179917Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2025-03-10T06:43:37.191665Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 2025-03-10T06:43:37.195822Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2025-03-10T06:43:37.195863Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2025-03-10T06:43:37.209862Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2025-03-10T06:43:37.210297Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active. 2025-03-10T06:43:37.210301Z 0 [Note] InnoDB: 32 n

CodeGolang
  • 粉丝: 182
上传资源 快速赚钱