class RecordBatchingSink[+B <: RecordBatch] extends PartitionGroupingSink
A Sink that uses a specified RecordBatcher to construct record batches and commit them to a specified RecordBatchStorage once they are ready. It is a partition grouping sink, meaning records from each partition group are added to different batches and processed separately. Batch commits to storage are queued up and performed asynchronously in the background for each partition group. The commit queue size is configurable and commits block if the queues get full.
- B
Type of record batches.
- Alphabetic
- By Inheritance
- RecordBatchingSink
- PartitionGroupingSink
- Logging
- Sink
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new RecordBatchingSink(recordBatcher: RecordBatcher[B], batchStorage: RecordBatchStorage[B], batchCommitQueueSize: Int, partitionGrouping: (TopicPartition) => String, retryPolicy: Policy)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- final def assignPartitions(partitions: Set[TopicPartition]): Map[TopicPartition, Option[StreamPosition]]
Assigns a set of partitions to the sink and returns their committed stream positions.
Assigns a set of partitions to the sink and returns their committed stream positions. Called during the initial partitions subscription and on subsequent rebalance events.
- partitions
Set of partitions assigned.
- returns
Committed stream position map for all the assigned partitions.
- Definition Classes
- PartitionGroupingSink → Sink
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def close(): Unit
Closes all active sinkers.
Closes all active sinkers.
- Definition Classes
- PartitionGroupingSink → Sink
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def groupForPartition(topicPartition: TopicPartition): String
Maps a given topic partition to a group name.
Maps a given topic partition to a group name.
- returns
A group name, e.g.
"root"ortopicPartition.topic().
- Definition Classes
- RecordBatchingSink → PartitionGroupingSink
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def heartbeat(): Unit
Forwards the heartbeat to all sinkers.
Forwards the heartbeat to all sinkers.
- Definition Classes
- PartitionGroupingSink → Sink
- def initialize(context: KafkaContext): Unit
Initializes the sink with a KafkaContext.
Initializes the sink with a KafkaContext.
- Definition Classes
- RecordBatchingSink → PartitionGroupingSink → Sink
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val kafkaContext: KafkaContext
- Attributes
- protected
- Definition Classes
- PartitionGroupingSink
- val log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def revokePartitions(partitions: Set[TopicPartition]): Map[TopicPartition, Option[StreamPosition]]
Revokes a set of partitions from the sink, called during rebalance events.
Revokes a set of partitions from the sink, called during rebalance events. Returns committed stream positions for partitions that need rewinding.
- partitions
Set of partitions revoked.
- returns
Committed stream position map for partitions that need rewinding.
- Definition Classes
- PartitionGroupingSink → Sink
- final def sinkerForPartitionGroup(groupName: String, groupPartitions: Set[TopicPartition]): PartitionGroupSinker
Creates a new instance of a PartitionGroupSinker for a given partition group.
Creates a new instance of a PartitionGroupSinker for a given partition group. These instances are closed and re-created during rebalance events.
- groupName
Name of the partition group.
- returns
A new instance of a sinker for the given group.
- Definition Classes
- RecordBatchingSink → PartitionGroupingSink
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def write(record: StreamRecord): Unit
Forwards a consumer record to the correct partition sinker.
Forwards a consumer record to the correct partition sinker.
- record
Stream record to write.
- Definition Classes
- PartitionGroupingSink → Sink