trait Sink extends AnyRef
Base trait that describes Kafka consumer record sinking to some underlying storage. Implementers need to define committed position lookup for a list of assigned partitions which are then used to rewind the streams. They are also responsible for implementing offset committing.
Implementers can chose to use Kafka itself for offset storage by using the provided KafkaContext, they can also store/retrieve offsets to/from the storage itself. The delivery guarantees of a sink depend on the offset management, e.g. storing offsets atomically with data guarantees exactly-once storage, whereas using only Kafka for offset storage will usually result in either at-least-once of at-most-once semantics.
- Alphabetic
- By Inheritance
- Sink
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def assignPartitions(partitions: Set[TopicPartition]): Map[TopicPartition, Option[StreamPosition]]
Assigns a set of partitions to the sink and returns their committed stream positions.
Assigns a set of partitions to the sink and returns their committed stream positions. Called during the initial partitions subscription and on subsequent rebalance events.
- partitions
Set of partitions assigned.
- returns
Committed stream position map for all the assigned partitions.
- abstract def close(): Unit
Closes the sink and performs any necessary clean up.
- abstract def heartbeat(): Unit
Notifies the sink that record consumption is still active, called when no records get polled from Kafka.
Notifies the sink that record consumption is still active, called when no records get polled from Kafka. Gives the sink an opportunity to perform flushing when working with very low traffic streams.
- abstract def initialize(kafkaContext: KafkaContext): Unit
Initializes the sink with a KafkaContext.
Initializes the sink with a KafkaContext.
- kafkaContext
The context provided by the source, can be used for offset management in Kafka.
- abstract def revokePartitions(partitions: Set[TopicPartition]): Map[TopicPartition, Option[StreamPosition]]
Revokes a set of partitions from the sink, called during rebalance events.
Revokes a set of partitions from the sink, called during rebalance events. Returns committed stream positions for partitions that need rewinding.
- partitions
Set of partitions revoked.
- returns
Committed stream position map for partitions that need rewinding.
- abstract def write(record: StreamRecord): Unit
Writes a record to the underlying storage.
Writes a record to the underlying storage.
- record
Stream record to write.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()