com.adform.streamloader.vertica
InRowOffsetVerticaFileStorage
Companion object InRowOffsetVerticaFileStorage
class InRowOffsetVerticaFileStorage extends InDataOffsetBatchStorage[InRowOffsetVerticaFileRecordBatch] with Logging
A Vertica storage implementation, stores offsets in rows of data. Queries Vertica upon initialization in order to retrieve committed stream positions.
Users should keep in mind that the data usage in the licensing audit is calculated treating everything as a string, thus storing the topic, partition and offset next to each row might be very expensive licensing-wise. For a cheaper alternative see the ExternalOffsetVerticaFileStorage.
- Alphabetic
- By Inheritance
- InRowOffsetVerticaFileStorage
- InDataOffsetBatchStorage
- Logging
- RecordBatchStorage
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new InRowOffsetVerticaFileStorage(dbDataSource: DataSource, table: String, topicColumnName: String, partitionColumnName: String, offsetColumnName: String, watermarkColumnName: String)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def commitBatch(batch: InRowOffsetVerticaFileRecordBatch): Unit
Stores a given batch to storage and commits offsets, preferably in a single atomic transaction.
Stores a given batch to storage and commits offsets, preferably in a single atomic transaction.
- Definition Classes
- InDataOffsetBatchStorage → RecordBatchStorage
- def commitBatchWithOffsets(batch: InRowOffsetVerticaFileRecordBatch): Unit
Stores a given batch to storage together with the offsets.
Stores a given batch to storage together with the offsets.
- Definition Classes
- InRowOffsetVerticaFileStorage → InDataOffsetBatchStorage
- def committedPositions(topicPartitions: Set[TopicPartition]): Map[TopicPartition, Option[StreamPosition]]
Gets the latest committed stream positions for the given partitions where streams should be sought to, i.e.
Gets the latest committed stream positions for the given partitions where streams should be sought to, i.e. this should be the last stored offset + 1.
- Definition Classes
- InRowOffsetVerticaFileStorage → RecordBatchStorage
- def committedPositions(connection: Connection): Map[TopicPartition, StreamPosition]
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def initialize(context: KafkaContext): Unit
Initializes the storage with a Kafka context, which can be used to lookup/commit offsets, if needed.
Initializes the storage with a Kafka context, which can be used to lookup/commit offsets, if needed.
- Definition Classes
- RecordBatchStorage
- def isBatchCommitted(batch: InRowOffsetVerticaFileRecordBatch): Boolean
Checks whether a given batch was successfully committed to storage by comparing committed positions with the record ranges in the batch.
Checks whether a given batch was successfully committed to storage by comparing committed positions with the record ranges in the batch.
- batch
Batch to check.
- returns
Whether the batch is successfully stored.
- Definition Classes
- RecordBatchStorage
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val kafkaContext: KafkaContext
- Attributes
- protected
- Definition Classes
- RecordBatchStorage
- val log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def recover(topicPartitions: Set[TopicPartition]): Unit
Performs any needed recovery upon startup, e.g.
Performs any needed recovery upon startup, e.g. rolling back or completing transactions. Can fail, users should handle any possible exceptions.
- Definition Classes
- InDataOffsetBatchStorage → RecordBatchStorage
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()