org.apache.flume.source.kafka

## Class KafkaSource

- java.lang.Object
- - org.apache.flume.source.BasicSourceSemantics
  - - org.apache.flume.source.AbstractPollableSource
    - - org.apache.flume.source.kafka.KafkaSource

- All Implemented Interfaces:
    Configurable, LifecycleAware, NamedComponent, PollableSource, Source

      public class KafkaSource
      extends AbstractPollableSource
      implements Configurable

  A Source for Kafka which reads messages from kafka topics. `kafka.bootstrap.servers:` A comma separated list of host:port pairs to use for establishing the initial connection to the Kafka cluster. For example host1:port1,host2:port2,... Required for kafka.

  `kafka.consumer.group.id:` the group ID of consumer group. Required

  `kafka.topics:` the topic list separated by commas to consume messages from. Required

  `maxBatchSize:` Maximum number of messages written to Channel in one batch. Default: 1000

  `maxBatchDurationMillis:` Maximum number of milliseconds before a batch (of any size) will be written to a channel. Default: 1000

  `kafka.consumer.*:` Any property starting with "kafka.consumer" will be passed to the kafka consumer So you can use any configuration supported by Kafka 0.9.0.X `useFlumeEventFormat:` Reads events from Kafka Topic as an Avro FlumeEvent. Used in conjunction with useFlumeEventFormat (Kafka Sink) or parseAsFlumeEvent (Kafka Channel)

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `class` | `KafkaSource.Subscriber<T>` This class is a helper to subscribe for topics by using different strategies

    - ### Nested classes/interfaces inherited from interface org.apache.flume.PollableSource

      `PollableSource.Status`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `KafkaSource()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `doConfigure(Context context)` We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1.
    `protected PollableSource.Status` | `doProcess()`
    `protected void` | `doStart()`
    `protected void` | `doStop()`

    - ### Methods inherited from class org.apache.flume.source.AbstractPollableSource

      `configure, getBackOffSleepIncrement, getMaxBackOffSleepInterval, process`

    - ### Methods inherited from class org.apache.flume.source.BasicSourceSemantics

      `getChannelProcessor, getLifecycleState, getName, getStartException, isStarted, setChannelProcessor, setLifecycleState, setName, start, stop, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

    - ### Methods inherited from interface org.apache.flume.conf.Configurable

      `configure`

    - ### Methods inherited from interface org.apache.flume.Source

      `getChannelProcessor, setChannelProcessor`

    - ### Methods inherited from interface org.apache.flume.lifecycle.LifecycleAware

      `getLifecycleState, start, stop`

    - ### Methods inherited from interface org.apache.flume.NamedComponent

      `getName, setName`

- - ### Constructor Detail

    - #### KafkaSource

          public KafkaSource()

  - ### Method Detail

    - #### doProcess

          protected PollableSource.Status doProcess()
                                             throws EventDeliveryException

      Specified by:
        `doProcess` in class `AbstractPollableSource`
      Throws:
        `EventDeliveryException`

    - #### doConfigure

          protected void doConfigure(Context context)
                              throws FlumeException

      We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1. Generate a properties object with some static defaults that can be overridden if corresponding properties are specified 2. We add the configuration users added for Kafka (parameters starting with kafka.consumer and must be valid Kafka Consumer properties 3. Add source level properties (with no prefix)
      Specified by:
        `doConfigure` in class `BasicSourceSemantics`
      Parameters:
        `context` -
      Throws:
        `FlumeException`

    - #### doStart

          protected void doStart()
                          throws FlumeException

      Specified by:
        `doStart` in class `BasicSourceSemantics`
      Throws:
        `FlumeException`

    - #### doStop

          protected void doStop()
                         throws FlumeException

      Specified by:
        `doStop` in class `BasicSourceSemantics`
      Throws:
        `FlumeException`

Copyright © 2009-2016 Apache Software Foundation. All Rights Reserved.