class RecordBatchingSinker[B <: RecordBatch] extends PartitionGroupSinker with Logging with Metrics
A PartitionGroupSinker that accumulates records to batches and stores them to some storage once ready.
- B
Type of record batches persisted to storage.
- Self Type
- RecordBatchingSinker[B]
- Alphabetic
- By Inheritance
- RecordBatchingSinker
- Metrics
- Logging
- PartitionGroupSinker
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new RecordBatchingSinker(groupName: String, groupPartitions: Set[TopicPartition], recordBatcher: RecordBatcher[B], batchStorage: RecordBatchStorage[B], batchCommitQueueSize: Int, retryPolicy: Policy)
- groupName
The partition group name (used for metrics and logs).
- groupPartitions
Set of Kafka topic partitions to sink.
- recordBatcher
A record batcher to use.
- batchStorage
A storage to use.
- batchCommitQueueSize
Size of the batch commit queue, once full further attempts to store batches will block.
- retryPolicy
The retry policy to use for all retriable operations.
Type Members
- class AssignableGauge[T <: AnyRef] extends AnyRef
- Definition Classes
- Metrics
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def close(): Unit
Cleans up and closes the sinker.
Cleans up and closes the sinker.
- Definition Classes
- RecordBatchingSinker → PartitionGroupSinker
- def commitQueueSize: Int
- def createAssignableGauge[T <: AnyRef](name: String, tdf: ToDoubleFunction[T], tags: Seq[MetricTag] = Seq()): AssignableGauge[T]
- Attributes
- protected
- Definition Classes
- Metrics
- def createCounter(name: String, tags: Seq[MetricTag] = Seq()): Counter
- Attributes
- protected
- Definition Classes
- Metrics
- def createDistribution(name: String, tags: Seq[MetricTag] = Seq()): DistributionSummary
- Attributes
- protected
- Definition Classes
- Metrics
- def createGauge[T <: AnyRef](name: String, metric: T, tdf: ToDoubleFunction[T], tags: Seq[MetricTag] = Seq()): Gauge
- Attributes
- protected
- Definition Classes
- Metrics
- def createTimer(name: String, tags: Seq[MetricTag] = Seq(), maxDuration: Duration = null): Timer
- Attributes
- protected
- Definition Classes
- Metrics
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- val groupName: String
Name of the partition group, can be used in metrics and logs to differentiate between group sinkers.
Name of the partition group, can be used in metrics and logs to differentiate between group sinkers.
- Definition Classes
- RecordBatchingSinker → PartitionGroupSinker
- val groupPartitions: Set[TopicPartition]
The Kafka topic partitions that this sinker is responsible for.
The Kafka topic partitions that this sinker is responsible for.
- Definition Classes
- RecordBatchingSinker → PartitionGroupSinker
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def heartbeat(): Unit
Notifies the sinker that record consumption is still active.
Notifies the sinker that record consumption is still active. Gives the sinker an opportunity to perform flushing with very low traffic streams.
- Definition Classes
- RecordBatchingSinker → PartitionGroupSinker
- def initialize(context: KafkaContext): Map[TopicPartition, Option[StreamPosition]]
Initializes the sinker and returns the stream positions where topic partitions should be reset to before loading.
Initializes the sinker and returns the stream positions where topic partitions should be reset to before loading. Should be called once before any subsequent calls to write. This is most likely a blocking call since it queries Kafka/storage for offsets.
- returns
The initial loader positions in the owned topic partitions. If no position is returned the position will not be reset explicitly, meaning that the consumer will either reset it to the earliest/latest based on the configuration value of
auto.offset.reset, or will reset to the offset stored in Kafka, if any.
- Definition Classes
- RecordBatchingSinker → PartitionGroupSinker
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- var kafkaContext: KafkaContext
- Attributes
- protected
- val log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- val metricsRoot: String
A common prefix for all created metrics.
A common prefix for all created metrics.
- Definition Classes
- RecordBatchingSinker → Metrics
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def removeMeters(meters: Meter*): Unit
- Attributes
- protected
- Definition Classes
- Metrics
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def write(record: StreamRecord): Unit
Writes a given stream record to storage.
Writes a given stream record to storage.
Calling this method does not ensure that the record will be flushed to storage, e.g. the sinker might implement batching thus delaying the actual storage.
- Definition Classes
- RecordBatchingSinker → PartitionGroupSinker