티스토리 뷰

공부

[Spark] streaming initial offset step by step

seunggabi 승가비 2020. 9. 12. 13:48
--conf spark.streaming.blockInterval=200 \
--conf spark.streaming.receiver.writeAheadLog.enable=true \
--conf spark.streaming.backpressure.enabled=true \
--conf spark.streaming.backpressure.pid.minRate=10 \
--conf spark.streaming.receiver.maxRate=100 \
--conf spark.streaming.kafka.maxRatePerPartition=100 \
--conf spark.streaming.backpressure.initialRate=30 \

https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/

 

24/7 Spark Streaming on YARN in Production - inovex Blog

We have been running Spark Streaming on Apache Hadoop™ YARN in production for close to a year now. This is what we learned.

www.inovex.de

 

댓글
댓글쓰기 폼