You can increase the batch size by changing "tridentKafkaConfig.fetchSizeBytes" property.
Also, batch size is related with number of brokers and number of partitions.
For example, if you have 2 brokers and 3 partitions for each broker that means the total count of partition is 6.
y this way, the batch size equals to tridentKafkaConfig.fetchSizeBytes X total partition count.
if we assume that the tridentKafkaConfig.fetchSizeBytes is 1024X1024, the batch size equals to 6 MB.(3x2x1024x1024)bytes
相当于从kafka里面是批量读取的，然后形成storm中的一个batch
而且spout里面有一个buffer，应该也算是一个缓存，将正在处理的一部分数据，存在内存中，直到处理完
网友评论