Eager Rebalance
Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1) consuming it. When a second consumer appears (consumer2) the rebalance task triggers for both of them (consumer1 gets an event, consumer2 does the initial rebalance). Now consumer1 closes all the existing connections (even those that will be reopened soon) and releases the partition ownership in Zookeeper for all 10 partitions.

Then it runs the partition assignment algorithm and decides what partitions should be claimed and claims the partition ownership in Zookeeper again. If the claim was successful consumer1 starts fetching his new partitions.

Meanwhile consumer2 runs the partition assignment algorithm as well and tries to claim his partitions in Zookeeper as well. Claim will succeed only when consumer1 releases the ownership on these partitions. When the claim is successful consumer2 starts fetching, or if it fails to claim partitions within a given amount of retries you get a rebalance failed after n retries exception.

As you noticed instead of just closing connections and releasing ownership for partitions consumer1 does not own anymore, it unnecessarily closes ALL his connections and restarts with just a lower amount of partitions. The same story with adding partitions (when we consume by a wildcard filter and new topic appears) – ALL connections are closed and then opened again instead of just opening new ones.

What is Disadvantage of Above?
All consumers:

1.Stop consuming in order to give up their partition ownership
2.Re-join the group via the JoinGroup request
3.Receive a brand new partition assignment via the SyncGroup request, only once the rebalance finishes

There is a short window of unavailability for the entire consumer group between steps 1) and 3) – a “stop the world” event.

Three things may happen which affect the availability of data during rebalance

1. happy path: how fast the consumers can join back
2. not-so-happy path: the session_timeout_ms
3. worst case: until your consumers stabilize

If your rebalance doesn’t execute successfully then rebalance would be restart again
If your consumer doesn’t join the group at all in session_timeout_ms, the group will complete the rebalance without them.When it does catch up and join – a new rebalance will start.
If your consumer starts the rebalance dance but doesn’t complete it in session_timeout_ms – the group will abort the rebalance, kick them out of the group, and begin a new rebalance.

And when it restarts back up and joins – a new rebalance will start again.

Besides the trickiness of the timeout during a rebalance, the basic cases that can start a new rebalance are:
• if any consumer fails.
• if any consumer restarts.
• if a new consumer joins the group.

So we both have:
1. plenty of common cases that trigger rebalances
2. tricky cases that can disrupt ongoing rebalances

Consumer group rebalance is nothing more than a partition reassignment between the consumers. If at all times there’s only one consumer joining/leaving the group.It does not make much sense to pause all the other consumers when they are bound to get the exact same partitions after the rebalance finishes.