Type Members

1.  trait PartitionGroupSinker extends AnyRef

   Base sinker responsible for loading a fixed set of Kafka topic partitions to persistent storage.

   Base sinker responsible for loading a fixed set of Kafka topic partitions to persistent storage. Implementers must define committed stream position lookup from the underlying storage upon initialization.

2.  trait PartitionGroupingSink extends Sink with Logging

   An abstract sink that implements partition assignment/revocation by grouping partitions to sets using a user defined function and delegates the actual data sinking to instances of PartitionGroupSinker, which operate with fixed sets of partitions and thus do not need to worry about rebalance events.

   An abstract sink that implements partition assignment/revocation by grouping partitions to sets using a user defined function and delegates the actual data sinking to instances of PartitionGroupSinker, which operate with fixed sets of partitions and thus do not need to worry about rebalance events.

   Implementers must define a partition grouping, which is a mapping from a topic partition to a group name string, the simplest grouping would simply map all partitions to a "root" group, a more elaborate grouping could create separate groups for smaller and larger topics, each group receiving a separate PartitionGroupSinker. E.g., if the sinker writes all data to a file, you would end up with separate files for smaller and larger topics.

3.  abstract class RewindingPartitionGroupSinker extends WrappedPartitionGroupSinker with Logging

   A wrapper sinker that rewinds the streams back by a given interval during initialization.

   A wrapper sinker that rewinds the streams back by a given interval during initialization. Can be used to implement stateful sinks that need to "warm-up" before starting actual writing, e.g. in order to implement record de-duplication one can rewind the streams in order to build-up a cache. Implementers need to override the method for "touching" rewound records, once the sinker catches up all new records are simply passed down to the base sinker.

   Rewinding by an offset range is straightforward as we simply subtract, a caveat here is that we can't determine the rewound watermark, so we retain it. Rewinding by watermark is done by calling offsetForTimes in Kafka, which can in principle return a result that is not exactly consistent with the watermark calculated by the stream loader. Either way the rewinding should be considered to be done on a best-effort basis.

4.  trait Sink extends AnyRef

   Base trait that describes Kafka consumer record sinking to some underlying storage.

   Base trait that describes Kafka consumer record sinking to some underlying storage. Implementers need to define committed position lookup for a list of assigned partitions which are then used to rewind the streams. They are also responsible for implementing offset committing.

   Implementers can chose to use Kafka itself for offset storage by using the provided KafkaContext, they can also store/retrieve offsets to/from the storage itself. The delivery guarantees of a sink depend on the offset management, e.g. storing offsets atomically with data guarantees exactly-once storage, whereas using only Kafka for offset storage will usually result in either at-least-once of at-most-once semantics.

5.  abstract class WrappedPartitionGroupSinker extends PartitionGroupSinker

   A PartitionGroupSinker that wraps another sinker instance and by default forwards all actions to it.

   A PartitionGroupSinker that wraps another sinker instance and by default forwards all actions to it. Used for implementing other wrapper sinks in order to avoid boilerplate.

6.  abstract class WrappedSink[S <: Sink] extends Sink

   A Sink that wraps another sink instance and by default forwards all operations to it.

   A Sink that wraps another sink instance and by default forwards all operations to it. Used for implementing concrete wrapping sinks in order to avoid boilerplate.

   S

   Type of sink being wrapped.