org.apache.flume.sink.kafka

## Class KafkaSink

- java.lang.Object
- - org.apache.flume.sink.AbstractSink
  - - org.apache.flume.sink.kafka.KafkaSink

- All Implemented Interfaces:
    Configurable, LifecycleAware, NamedComponent, Sink

      public class KafkaSink
      extends AbstractSink
      implements Configurable

  A Flume Sink that can publish messages to Kafka. This is a general implementation that can be used with any Flume agent and a channel. The message can be any event and the key is a string that we read from the header For use of partitioning, use an interceptor to generate a header with the partition key

  Mandatory properties are: brokerList -- can be a partial list, but at least 2 are recommended for HA

  however, any property starting with "kafka." will be passed along to the Kafka producer Read the Kafka producer documentation to see which configurations can be used

  Optional properties topic - there's a default, and also - this can be in the event header if you need to support events with different topics batchSize - how many messages to process in one batch. Larger batches improve throughput while adding latency. requiredAcks -- 0 (unsafe), 1 (accepted by at least one broker, default), -1 (accepted by all brokers) useFlumeEventFormat - preserves event headers when serializing onto Kafka

  header properties (per event): topic key

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from interface org.apache.flume.Sink

      `Sink.Status`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `KafkaSink()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `configure(Context context)` We configure the sink and generate properties for the Kafka Producer Kafka producer properties is generated as follows: 1.
    `int` | `getBatchSize()`
    `protected Properties` | `getKafkaProps()`
    `String` | `getTopic()`
    `Sink.Status` | `process()` Requests the sink to attempt to consume data from attached channel
    `void` | `start()` Starts a service or component.
    `void` | `stop()` Stops a service or component.

    - ### Methods inherited from class org.apache.flume.sink.AbstractSink

      `getChannel, getLifecycleState, getName, setChannel, setName, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Constructor Detail

    - #### KafkaSink

          public KafkaSink()

  - ### Method Detail

    - #### getTopic

          public String getTopic()

    - #### getBatchSize

          public int getBatchSize()

    - #### process

          public Sink.Status process()
                              throws EventDeliveryException

      Description copied from interface: `Sink`

      Requests the sink to attempt to consume data from attached channel

      Note: This method should be consuming from the channel within the bounds of a Transaction. On successful delivery, the transaction should be committed, and on failure it should be rolled back.

      Specified by:
        `process` in interface `Sink`
      Returns:
        READY if 1 or more Events were successfully delivered, BACKOFF if no data could be retrieved from the channel feeding this sink
      Throws:
        `EventDeliveryException` - In case of any kind of failure to deliver data to the next hop destination.

    - #### start

          public void start()

      Description copied from interface: `LifecycleAware`

      Starts a service or component.

      Implementations should determine the result of any start logic and effect the return value of `LifecycleAware.getLifecycleState()` accordingly.

      Specified by:
        `start` in interface `LifecycleAware`
      Overrides:
        `start` in class `AbstractSink`

    - #### stop

          public void stop()

      Description copied from interface: `LifecycleAware`

      Stops a service or component.

      Implementations should determine the result of any stop logic and effect the return value of `LifecycleAware.getLifecycleState()` accordingly.

      Specified by:
        `stop` in interface `LifecycleAware`
      Overrides:
        `stop` in class `AbstractSink`

    - #### configure

          public void configure(Context context)

      We configure the sink and generate properties for the Kafka Producer Kafka producer properties is generated as follows: 1. We generate a properties object with some static defaults that can be overridden by Sink configuration 2. We add the configuration users added for Kafka (parameters starting with .kafka. and must be valid Kafka Producer properties 3. We add the sink's documented parameters which can override other properties
      Specified by:
        `configure` in interface `Configurable`
      Parameters:
        `context` -

    - #### getKafkaProps

          protected Properties getKafkaProps()

Copyright © 2009-2016 Apache Software Foundation. All Rights Reserved.