大數(shù)據(jù)培訓Consumer配置信息

屬性

默認值

描述

group.id

Consumer的組ID，相同goup.id的consumer屬于同一個組。

zookeeper.connect

Consumer的zookeeper連接串，要和broker的配置一致。

consumer.id

null

如果不設置會自動生成。

socket.timeout.ms

30 * 1000

網(wǎng)絡請求的socket超時時間。實際超時時間由max.fetch.wait + socket.timeout.ms 確定。

socket.receive.buffer.bytes

64 * 1024

The socket receive buffer for network requests.

fetch.message.max.bytes

1024 * 1024

查詢topic-partition時允許的最大消息大小。consumer會為每個partition緩存此大小的消息到內(nèi)存，因此，這個參數(shù)可以控制consumer的內(nèi)存使用量。這個值應該至少比server允許的最大消息大小大，以免producer發(fā)送的消息大于consumer允許的消息。

num.consumer.fetchers

1

The number fetcher threads used to fetch data.

auto.commit.enable

true

如果此值設置為true，consumer會周期性的把當前消費的offset值保存到zookeeper。當consumer失敗重啟之后將會使用此值作為新開始消費的值。

auto.commit.interval.ms

60 * 1000

Consumer提交offset值到zookeeper的周期。

queued.max.message.chunks

2

用來被consumer消費的message chunks 數(shù)量， 每個chunk可以緩存fetch.message.max.bytes大小的數(shù)據(jù)量。

auto.commit.interval.ms

60 * 1000

Consumer提交offset值到zookeeper的周期。

queued.max.message.chunks

2

用來被consumer消費的message chunks 數(shù)量， 每個chunk可以緩存fetch.message.max.bytes大小的數(shù)據(jù)量。

fetch.min.bytes

1

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request.

fetch.wait.max.ms

100

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy fetch.min.bytes.

rebalance.backoff.ms

2000

Backoff time between retries during rebalance.

refresh.leader.backoff.ms

200

Backoff time to wait before trying to determine the leader of a partition that has just lost its leader.

auto.offset.reset

largest

What to do when there is no initial offset in ZooKeeper or if an offset is out of range ;smallest : automatically reset the offset to the smallest offset; largest : automatically reset the offset to the largest offset;anything else: throw exception to the consumer

consumer.timeout.ms

-1

若在指定時間內(nèi)沒有消息消費，consumer將會拋出異常。

exclude.internal.topics

true

Whether messages from internal topics (such as offsets) should be exposed to the consumer.

zookeeper.session.timeout.ms

6000

ZooKeeper session timeout. If the consumer fails to heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur.

zookeeper.connection.timeout.ms

6000

The max time that the client waits while establishing a connection to zookeeper.

zookeeper.sync.time.ms

2000

How far a ZK follower can be behind a ZK leader