abstract class RewindingPartitionGroupSinker extends WrappedPartitionGroupSinker with Logging
A wrapper sinker that rewinds the streams back by a given interval during initialization. Can be used to implement stateful sinks that need to "warm-up" before starting actual writing, e.g. in order to implement record de-duplication one can rewind the streams in order to build-up a cache. Implementers need to override the method for "touching" rewound records, once the sinker catches up all new records are simply passed down to the base sinker.
Rewinding by an offset range is straightforward as we simply subtract, a caveat here is that we can't determine the rewound watermark, so we retain it. Rewinding by watermark is done by calling offsetForTimes in Kafka, which can in principle return a result that is not exactly consistent with the watermark calculated by the stream loader. Either way the rewinding should be considered to be done on a best-effort basis.
- Alphabetic
- By Inheritance
- RewindingPartitionGroupSinker
- Logging
- WrappedPartitionGroupSinker
- PartitionGroupSinker
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new RewindingPartitionGroupSinker(baseSinker: PartitionGroupSinker, rewindInterval: StreamInterval)
- baseSinker
A base sinker to wrap.
- rewindInterval
A stream interval to rewind backwards.
Abstract Value Members
- abstract def touchRewoundRecord(record: StreamRecord): Unit
Process a given stream record that is already written by the base sink, but is now rewound back during initialization for warm-up.
Process a given stream record that is already written by the base sink, but is now rewound back during initialization for warm-up.
- record
Stream record to "touch".
- Attributes
- protected
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def close(): Unit
Cleans up and closes the sinker.
Cleans up and closes the sinker.
- Definition Classes
- WrappedPartitionGroupSinker → PartitionGroupSinker
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- val groupName: String
Name of the partition group, can be used in metrics and logs to differentiate between group sinkers.
Name of the partition group, can be used in metrics and logs to differentiate between group sinkers.
- Definition Classes
- WrappedPartitionGroupSinker → PartitionGroupSinker
- val groupPartitions: Set[TopicPartition]
The Kafka topic partitions that this sinker is responsible for.
The Kafka topic partitions that this sinker is responsible for.
- Definition Classes
- WrappedPartitionGroupSinker → PartitionGroupSinker
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def heartbeat(): Unit
Notifies the sinker that record consumption is still active.
Notifies the sinker that record consumption is still active. Gives the sinker an opportunity to perform flushing with very low traffic streams.
- Definition Classes
- WrappedPartitionGroupSinker → PartitionGroupSinker
- def initialize(kafkaContext: KafkaContext): Map[TopicPartition, Option[StreamPosition]]
Initializes the sinker and returns the stream positions where topic partitions should be reset to before loading.
Initializes the sinker and returns the stream positions where topic partitions should be reset to before loading. Should be called once before any subsequent calls to write. This is most likely a blocking call since it queries Kafka/storage for offsets.
- kafkaContext
Kafka context to use when looking up / committing offsets to Kafka, if needed.
- returns
The initial loader positions in the owned topic partitions. If no position is returned the position will not be reset explicitly, meaning that the consumer will either reset it to the earliest/latest based on the configuration value of
auto.offset.reset, or will reset to the offset stored in Kafka, if any.
- Definition Classes
- RewindingPartitionGroupSinker → WrappedPartitionGroupSinker → PartitionGroupSinker
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def write(record: StreamRecord): Unit
Checks whether the given record is a "rewound" record, i.e.
Checks whether the given record is a "rewound" record, i.e. is already committed by the base sink and if so only "touches" it, otherwise passes it further down to the base sink.
- Definition Classes
- RewindingPartitionGroupSinker → WrappedPartitionGroupSinker → PartitionGroupSinker