恰好一次发送和事务消息(译)
Kafka提供“至少一次”交付语义, 这意味着发送的消息可以传送一次或多次. 人们真正想要的是“一次”语义,因为重复的消息没有被传递。
普遍地发声重复消息的情况有两种:
- 如果客户端尝试向集群发送消息并获取网络错误, 则重试可能会导致重复. 如果在发送消息之前发生网络错误, 则不会发生重复. 但是, 如果在将消息附加到日志之后发生网络错误, 但在将响应发送给发件人之前, 发件人将不知道发生了什么. 唯一的选择是重试和冒险重复或放弃并声明消息丢失。
- 如果客户端尝试向集群发送消息并获取网络错误, 则重试可能会导致重复. 如果在发送消息之前发生网络错误, 则不会发生重复. 但是, 如果在将消息附加到日志之后发生网络错误, 但在将响应发送给发件人之前, 发件人将不知道发生了什么. 唯一的选择是重试和冒险重复或放弃并声明消息丢失。
第二种情况可以通过使用Kafka提供的偏移量由消费者处理. 他们可以将偏移量与其输出进行存储, 然后确保新消费者始终从最后存储的偏移量中提取. 或者, 他们可以使用偏移量作为一种关键字, 并使用它来对其输出的任何最终目标系统进行重复数据删除。
Producer API改动
KafkaProducer.java
public interface Producer<K,V> extends Closeable {
/**
* Needs to be called before any of the other transaction methods. Assumes that
* the transactional.id is specified in the producer configuration.
*
* This method does the following:
* 1. Ensures any transactions initiated by previous instances of the producer
* are completed. If the previous instance had failed with a transaction in
* progress, it will be aborted. If the last transaction had begun completion,
* but not yet finished, this method awaits its completion.
* 2. Gets the internal producer id and epoch, used in all future transactional
* messages issued by the producer.
*
* @throws IllegalStateException if the TransactionalId for the producer is not set
* in the configuration.
*/
void initTransactions() throws IllegalStateException;
/**
* Should be called before the start of each new transaction.
*
* @throws ProducerFencedException if another producer is with the same
* transactional.id is active.
*/
void beginTransaction() throws ProducerFencedException;
/**
* Sends a list of consumed offsets to the consumer group coordinator, and also marks
* those offsets as part of the current transaction. These offsets will be considered
* consumed only if the transaction is committed successfully.
*
* This method should be used when you need to batch consumed and produced messages
* together, typically in a consume-transform-produce pattern.
*
* @throws ProducerFencedException if another producer is with the same
* transactional.id is active.
*/
void sendOffsetsToTransaction(Map<TopicPartition, OffsetAndMetadata> offsets,
String consumerGroupId) throws ProducerFencedException;
/**
* Commits the ongoing transaction.
*
* @throws ProducerFencedException if another producer is with the same
* transactional.id is active.
*/
void commitTransaction() throws ProducerFencedException;
/**
* Aborts the ongoing transaction.
*
* @throws ProducerFencedException if another producer is with the same
* transactional.id is active.
*/
void abortTransaction() throws ProducerFencedException;
/**
* Send the given record asynchronously and return a future which will eventually contain the response information.
*
* @param record The record to send
* @return A future which will eventually contain the response information
*
*/
public Future<RecordMetadata> send(ProducerRecord<K, V> record);
/**
* Send a record and invoke the given callback when the record has been acknowledged by the server
*/
public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback);
}
OutOfSequenceException
如果broker检测到数据丢失,生产者将抛出OutOfOrderSequenceException。 换句话说,如果它接收到大于其预期的序列的序列号。 未来将返回此异常,并传递给回调(如果有)。 这是一个致命的异常,新的Producer方法如send,beginTransaction,commitTransaction等将会抛出IlegalStateException。