Packages

  • package root
    Definition Classes
    root
  • package com
    Definition Classes
    root
  • package adform
    Definition Classes
    com
  • package streamloader

    The entry point of the stream loader library is the StreamLoader class, which requires a KafkaSource and a Sink.

    The entry point of the stream loader library is the StreamLoader class, which requires a KafkaSource and a Sink. Once started it will subscribe to the provided topics and will start polling and sinking records. The sink has to be able to persist records and to look up committed offsets (technically this is optional, but without it there would be no way to provide any delivery guarantees). A large class of sinks are batch based, implemented as RecordBatchingSink. This sink accumulate batches of records using some RecordBatcher and once ready, stores them to some underlying RecordBatchStorage. A common type of batch is file based, i.e. a batcher might write records to a temporary file and once the file is full the sink commits the file to some underlying storage, such as a database or a distributed file system like HDFS.

    A sketch of the class hierarchy illustrating the main classes and interfaces can be seen below.



    For concrete storage implementations see the clickhouse, hadoop, s3 and vertica packages. They also contain more file builder implementations than just the CsvFileBuilder included in the core library.

    Definition Classes
    adform
  • package sink
    Definition Classes
    streamloader
  • package encoding
    Definition Classes
    sink
  • package binary
    Definition Classes
    encoding
  • ByteWriter
  • LittleEndianByteWriter

trait ByteWriter extends AnyRef

A base trait that writes bytes to some underlying storage.

Linear Supertypes
AnyRef, Any
Known Subclasses
RowBinaryClickHousePrimitiveTypeWriter, LittleEndianByteWriter, NativeVerticaPrimitiveTypeWriter
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ByteWriter
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract def writeByte(b: Int): Unit

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  9. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  13. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. def stringToBytes(s: String, maxBytes: Int): (Array[Byte], Int)

    Converts a given string to it's UTF-8 byte representation and truncates it at the character boundary if needed, so that it does not occupy more that the specified max bytes.

    Converts a given string to it's UTF-8 byte representation and truncates it at the character boundary if needed, so that it does not occupy more that the specified max bytes.

    s

    String to convert

    maxBytes

    Max bytes to return

    returns

    The full string byte representation and the number of bytes that fit into the max specified bytes.

    Attributes
    protected
  15. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  16. def toString(): String
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  18. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  19. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  20. def writeByteArray(bytes: Array[Byte]): Unit
  21. def writeByteArray(bytes: Array[Byte], maxLength: Int, truncate: Boolean = true): Unit

    Writes a specified number of bytes from a given array and stops once the limit is reached.

    Writes a specified number of bytes from a given array and stops once the limit is reached. Fails immediately if the byte array length exceeds length and truncate is false.

  22. def writeBytes(bytes: Int*): Unit
  23. def writeFixedByteArray(bytes: Array[Byte], length: Int, truncate: Boolean, padWith: Byte): Unit

    Writes bytes from the given array and either pads it to the required length or truncates it if the length of the array exceeds the required length.

    Writes bytes from the given array and either pads it to the required length or truncates it if the length of the array exceeds the required length. Fails immediately if the byte array length exceeds length and truncate is false.

  24. def writeFixedString(s: String, lengthBytes: Int, truncate: Boolean, padWith: Byte): Unit

    Writes the UTF-8 byte representation of the given string.

    Writes the UTF-8 byte representation of the given string. The string is truncated at the character boundary to fit into lengthBytes bytes, if truncate is true. Padding bytes are written to make the total number of bytes written equal to lengthBytes. Fails immediately if the byte array length exceeds length and truncate is false.

    Attributes
    protected

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AnyRef

Inherited from Any

Ungrouped