消费时offset被重置导致重复消费
这是实际使用时遇到的问题:kafka api的版本是0.10,发现有重复消费问题;检查log后发现在commit offset的时候发生超时。
Auto offset commit failed for group test: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
15:12:12.364 [main] WARN o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed for group test: Commit offsets failed with retriable exception. You should retry committing offsets.
看了Kafka的API文档,发现0.10中提供了新的配置max.poll.records:
The maximum number of records returned in a single call to poll().
type: int
default: 2147483647
如果生产端写入很快,消费端处理耗时。一个batch的处理时间大于session.timeout.ms,会导致session time out,引起offset commit失败。