如何删除.hive-staging文件

本文介绍如何修改Hive配置,将.staging目录更改为可定期清理的位置,通过设置crontab定时任务,实现每晚自动删除暂存数据,保持系统整洁。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

修改hive/conf目录下hive-site.xml配置文件

找到<name>hive.exec.stagingdir</name>

将value值修改到一个指定目录方便后期删除

<value>/tmp/staging/.staging</value>

保存分发配置

后面所有的.hive-staging数据均在上面的目录下

写一个crontab定时脚本每天夜里删除文件即可

hadoop fs -rm -r /tmp/staging/

Query ID = root_20250402142135_3ca1da7e-f563-4a4a-90af-d3f314ba82e6 Total jobs = 3 Launching Job 1 out of 3 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1743574293623_0001, Tracking URL = http://master:8088/proxy/application_1743574293623_0001/ Kill Command = /root/software/hadoop-3.3.0/bin/mapred job -kill job_1743574293623_0001 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-04-02 14:22:39,211 Stage-1 map = 0%, reduce = 0% 2025-04-02 14:22:54,895 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.48 sec 2025-04-02 14:23:02,452 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.82 sec MapReduce Total cumulative CPU time: 3 seconds 820 msec Ended Job = job_1743574293623_0001 Stage-4 is selected by condition resolver. Stage-3 is filtered out by condition resolver. Stage-5 is filtered out by condition resolver. Moving data to directory hdfs://master:9820/user/hive/warehouse/student/.hive-staging_hive_2025-04-02_14-21-35_316_3330971039988947438-1/-ext-10000 Loading data to table default.student MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.82 sec HDFS Read: 15213 HDFS Write: 241 SUCCESS Total MapReduce CPU Time Spent: 3 seconds 820 msec OK _col0 _col1 Time taken: 100.794 seconds
最新发布
04-03
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值