티스토리 뷰

728x90
--conf spark.streaming.blockInterval=200 \
--conf spark.streaming.receiver.writeAheadLog.enable=true \
--conf spark.streaming.backpressure.enabled=true \
--conf spark.streaming.backpressure.pid.minRate=10 \
--conf spark.streaming.receiver.maxRate=100 \
--conf spark.streaming.kafka.maxRatePerPartition=100 \
--conf spark.streaming.backpressure.initialRate=30 \

https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/

 

24/7 Spark Streaming on YARN in Production - inovex Blog

We have been running Spark Streaming on Apache Hadoop™ YARN in production for close to a year now. This is what we learned.

www.inovex.de

 

728x90

'공부' 카테고리의 다른 글

2020.09.15  (0) 2020.09.15
[Java] Arrays.asList UnsupportedOperationException  (0) 2020.09.12
[Kafka] Message Size over 1MB  (0) 2020.09.12
[Kafka] Producer TimeoutException: Expiring 1 record(s) & Network Exception  (0) 2020.09.12
[Sh] if  (0) 2020.08.28
댓글