python操作hbase如何快速录入100万数据,如何在python中将数据大容量加载到hbase

转载于 2021-03-26 12:29:49 发布 · 201 阅读

文章标签：

#python操作hbase如何快速录入100万数据

本文探讨了如何通过HBase的批量操作来提高数据写入效率，包括使用`Table`的批量API和设置合理的`batch_size`参数，以避免内存溢出和大批次数据发送问题。建议在应用中考虑安装Thrift服务器以支持这种高效数据处理方式。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

table = connection.table("mytable")

with table.batch(batch_size=1000) as b:

for i in range(1200):

b.put(b'row-%04d'.format(i), {

b'cf1:col1': b'v1',

b'cf1:col2': b'v2',

})As you may have imagined already, a Batch keeps all mutations in memory until the batch is sent, either by calling Batch.send() explicitly, or when the with block ends. This doesn’t work for applications that need to store huge amounts of data, since it may result in batches that are too big to send in one round-trip, or in batches that use too much memory. For these cases, the batch_size argument can be specified. The batch_size acts as a threshold: a Batch instance automatically sends all pending mutations when there are more than batch_size pending operations.

这需要在hbase之前安装一个Thrift服务器。只是个建议。在