Zookeeper学习8_关于ZookeeperConsumerConnector#rebalance-CSDN博客

本文详细介绍了Kafka中消费者组的负载均衡机制，包括如何为每个partition分配owner消费者，以及具体的实现细节。通过ZookeeperConsumerConnector#rebalance方法，确保了每个partition仅由一个consumer线程消费。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

consumer的负载均衡，每一个partition只会同时被consumer group里面的一台机器消费，称为owner，owner分配的逻辑，
1. For each topic T that Ci subscribes to
2. let PT be all partitions producing topic T
3. let CG be all consumers in the same group as Ci that consume topic T
4. sort PT (so partitions on the same broker are clustered together)
5. sort CG
6. let i be the index position of Ci in CG and let N = size(PT)/size(CG)
7. assign partitions from i*N to (i+1)*N - 1 to consumer Ci // i*N to i*N+N-1
8. remove current entries owned by Ci from the partition owner registry
9. add newly assigned partitions to the partition owner registry
(we may need to re-try this until the original partition owner releases its ownership)

具体实现在ZookeeperConsumerConnector#rebalance，

            for (consumerThreadId <- consumerThreadIdSet) {
            val myConsumerPosition = curConsumers.indexOf(consumerThreadId)
            assert(myConsumerPosition >= 0)
            val startPart = nPartsPerConsumer*myConsumerPosition + myConsumerPosition.min(nConsumersWithExtraPart)
            val nParts = nPartsPerConsumer + (if (myConsumerPosition + 1 > nConsumersWithExtraPart) 0 else 1)


            /**
             *   Range-partition the sorted partitions to consumers for better locality.
             *  The first few consumers pick up an extra partition, if any.
             */
            if (nParts <= 0)
              warn("No broker partitions consumed by consumer thread " + consumerThreadId + " for topic " + topic)
            else {
              for (i <- startPart until startPart + nParts) {
                val partition = curPartitions(i)
                info(consumerThreadId + " attempting to claim partition " + partition)


                // 每个partition只会由一条thread消费，不会有并发的问题
                addPartitionTopicInfo(currentTopicRegistry, topicDirs, partition, topic, consumerThreadId)


                // record the partition ownership decision
                partitionOwnershipDecision += ((topic, partition) -> consumerThreadId)
              }
            }
          }

最后当然也是存储在ZK上了，

/consumers/[group_id]/owners/[topic]/[broker_id-partition_id] --> consumer_node_id (ephemeral node)

资料节选自：http://www.sjsjw.com/104/000355MYM018196/

在使用过程中，遇到了一个错误，如标红的地方所示，故查找一下原因。