Packages

  • package root
    Definition Classes
    root
  • package com
    Definition Classes
    root
  • package adform
    Definition Classes
    com
  • package streamloader

    The entry point of the stream loader library is the StreamLoader class, which requires a KafkaSource and a Sink.

    The entry point of the stream loader library is the StreamLoader class, which requires a KafkaSource and a Sink. Once started it will subscribe to the provided topics and will start polling and sinking records. The sink has to be able to persist records and to look up committed offsets (technically this is optional, but without it there would be no way to provide any delivery guarantees). A large class of sinks are batch based, implemented as RecordBatchingSink. This sink accumulate batches of records using some RecordBatcher and once ready, stores them to some underlying RecordBatchStorage. A common type of batch is file based, i.e. a batcher might write records to a temporary file and once the file is full the sink commits the file to some underlying storage, such as a database or a distributed file system like HDFS.

    A sketch of the class hierarchy illustrating the main classes and interfaces can be seen below.



    For concrete storage implementations see the clickhouse, hadoop, s3 and vertica packages. They also contain more file builder implementations than just the CsvFileBuilder included in the core library.

    Definition Classes
    adform
  • package hadoop
    Definition Classes
    streamloader
  • package parquet
    Definition Classes
    hadoop
  • FileStaging
  • HadoopFileStorage
  • MultiFileStaging

class HadoopFileStorage[P] extends TwoPhaseCommitBatchStorage[PartitionedFileRecordBatch[P, FileRecordBatch], MultiFileStaging]

A Hadoop compatible file system based storage, most likely used for storing to HDFS. Stores files and commits offsets to Kafka in a two-phase transaction. The prepare/commit phases for storing a file consist of first uploading it to a staging path and later atomically moving it to the final destination path.

Linear Supertypes
TwoPhaseCommitBatchStorage[PartitionedFileRecordBatch[P, FileRecordBatch], MultiFileStaging], Metrics, Logging, RecordBatchStorage[PartitionedFileRecordBatch[P, FileRecordBatch]], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. HadoopFileStorage
  2. TwoPhaseCommitBatchStorage
  3. Metrics
  4. Logging
  5. RecordBatchStorage
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new HadoopFileStorage(hadoopFS: FileSystem, stagingDirectory: String, stagingFilePathFormatter: FilePathFormatter[P], destinationDirectory: String, destinationFilePathFormatter: FilePathFormatter[P])

Type Members

  1. class AssignableGauge[T <: AnyRef] extends AnyRef
    Definition Classes
    Metrics

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def commitBatch(batch: PartitionedFileRecordBatch[P, FileRecordBatch]): Unit

    Commits a given batch to storage.

    Commits a given batch to storage.

    Definition Classes
    TwoPhaseCommitBatchStorage → RecordBatchStorage
  7. final def committedPositions(topicPartitions: Set[TopicPartition]): Map[TopicPartition, Option[StreamPosition]]

    Gets the latest committed stream positions for the given partitions.

    Gets the latest committed stream positions for the given partitions.

    Definition Classes
    TwoPhaseCommitBatchStorage → RecordBatchStorage
  8. def createAssignableGauge[T <: AnyRef](name: String, tdf: ToDoubleFunction[T], tags: Seq[MetricTag] = Seq()): AssignableGauge[T]
    Attributes
    protected
    Definition Classes
    Metrics
  9. def createCounter(name: String, tags: Seq[MetricTag] = Seq()): Counter
    Attributes
    protected
    Definition Classes
    Metrics
  10. def createDistribution(name: String, tags: Seq[MetricTag] = Seq()): DistributionSummary
    Attributes
    protected
    Definition Classes
    Metrics
  11. def createGauge[T <: AnyRef](name: String, metric: T, tdf: ToDoubleFunction[T], tags: Seq[MetricTag] = Seq()): Gauge
    Attributes
    protected
    Definition Classes
    Metrics
  12. def createTimer(name: String, tags: Seq[MetricTag] = Seq(), maxDuration: Duration = null): Timer
    Attributes
    protected
    Definition Classes
    Metrics
  13. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  15. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def initialize(context: KafkaContext): Unit

    Initializes the storage with a Kafka context, which can be used to lookup/commit offsets, if needed.

    Initializes the storage with a Kafka context, which can be used to lookup/commit offsets, if needed.

    Definition Classes
    RecordBatchStorage
  18. def isBatchCommitted(batch: PartitionedFileRecordBatch[P, FileRecordBatch]): Boolean

    Checks whether a given batch was successfully committed to storage by comparing committed positions with the record ranges in the batch.

    Checks whether a given batch was successfully committed to storage by comparing committed positions with the record ranges in the batch.

    batch

    Batch to check.

    returns

    Whether the batch is successfully stored.

    Definition Classes
    RecordBatchStorage
  19. def isBatchStored(staging: MultiFileStaging): Boolean

    Checks whether a staged batch is actually stored, used during recovery.

    Checks whether a staged batch is actually stored, used during recovery.

    staging

    Batch staging information.

    returns

    Whether the batch is fully stored.

    Attributes
    protected
    Definition Classes
    HadoopFileStorage → TwoPhaseCommitBatchStorage
  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. val kafkaContext: KafkaContext
    Attributes
    protected
    Definition Classes
    RecordBatchStorage
  22. val log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  23. def metricsRoot: String

    A common prefix for all created metrics.

    A common prefix for all created metrics.

    Attributes
    protected
    Definition Classes
    TwoPhaseCommitBatchStorage → Metrics
  24. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  26. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  27. final def recover(topicPartitions: Set[TopicPartition]): Unit

    Performs any needed recovery upon startup, e.g.

    Performs any needed recovery upon startup, e.g. rolling back or completing transactions. Can fail, users should handle any possible exceptions.

    Definition Classes
    TwoPhaseCommitBatchStorage → RecordBatchStorage
  28. def removeMeters(meters: Meter*): Unit
    Attributes
    protected
    Definition Classes
    Metrics
  29. def stageBatch(batch: PartitionedFileRecordBatch[P, FileRecordBatch]): MultiFileStaging

    Stages a record batch to storage.

    Stages a record batch to storage.

    batch

    Record batch to store.

    returns

    Information about the staging.

    Attributes
    protected
    Definition Classes
    HadoopFileStorage → TwoPhaseCommitBatchStorage
  30. def storeBatch(staging: MultiFileStaging): Unit

    Finalizes storage of a staged record batch.

    Finalizes storage of a staged record batch.

    staging

    Batch staging information.

    Attributes
    protected
    Definition Classes
    HadoopFileStorage → TwoPhaseCommitBatchStorage
  31. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  32. def toString(): String
    Definition Classes
    AnyRef → Any
  33. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  34. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  35. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Ungrouped