trait PartitionGroupingSink extends Sink with Logging
An abstract sink that implements partition assignment/revocation by grouping partitions to sets using a user defined function and delegates the actual data sinking to instances of PartitionGroupSinker, which operate with fixed sets of partitions and thus do not need to worry about rebalance events.
Implementers must define a partition grouping, which is a mapping from a topic partition to a group name string,
the simplest grouping would simply map all partitions to a "root" group, a more elaborate grouping could create
separate groups for smaller and larger topics, each group receiving a separate PartitionGroupSinker.
E.g., if the sinker writes all data to a file, you would end up with separate files for smaller and larger topics.
- Alphabetic
- By Inheritance
- PartitionGroupingSink
- Logging
- Sink
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def groupForPartition(topicPartition: TopicPartition): String
Maps a given topic partition to a group name.
Maps a given topic partition to a group name.
- returns
A group name, e.g.
"root"ortopicPartition.topic().
- abstract def sinkerForPartitionGroup(groupName: String, partitions: Set[TopicPartition]): PartitionGroupSinker
Creates a new instance of a PartitionGroupSinker for a given partition group.
Creates a new instance of a PartitionGroupSinker for a given partition group. These instances are closed and re-created during rebalance events.
- groupName
Name of the partition group.
- partitions
Partitions in the group.
- returns
A new instance of a sinker for the given group.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- final def assignPartitions(partitions: Set[TopicPartition]): Map[TopicPartition, Option[StreamPosition]]
Assigns a set of partitions to the sink and returns their committed stream positions.
Assigns a set of partitions to the sink and returns their committed stream positions. Called during the initial partitions subscription and on subsequent rebalance events.
- partitions
Set of partitions assigned.
- returns
Committed stream position map for all the assigned partitions.
- Definition Classes
- PartitionGroupingSink → Sink
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def close(): Unit
Closes all active sinkers.
Closes all active sinkers.
- Definition Classes
- PartitionGroupingSink → Sink
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def heartbeat(): Unit
Forwards the heartbeat to all sinkers.
Forwards the heartbeat to all sinkers.
- Definition Classes
- PartitionGroupingSink → Sink
- def initialize(context: KafkaContext): Unit
Initializes the sink with a KafkaContext.
Initializes the sink with a KafkaContext.
- Definition Classes
- PartitionGroupingSink → Sink
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val kafkaContext: KafkaContext
- Attributes
- protected
- val log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def revokePartitions(partitions: Set[TopicPartition]): Map[TopicPartition, Option[StreamPosition]]
Revokes a set of partitions from the sink, called during rebalance events.
Revokes a set of partitions from the sink, called during rebalance events. Returns committed stream positions for partitions that need rewinding.
- partitions
Set of partitions revoked.
- returns
Committed stream position map for partitions that need rewinding.
- Definition Classes
- PartitionGroupingSink → Sink
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def write(record: StreamRecord): Unit
Forwards a consumer record to the correct partition sinker.
Forwards a consumer record to the correct partition sinker.
- record
Stream record to write.
- Definition Classes
- PartitionGroupingSink → Sink