티스토리 뷰

공부

[spark] Pyspark: Serialized task exceeds max allowed. Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values

승가비 2023. 2. 23. 00:30
728x90
--conf spark.rpc.message.maxSize=2047

https://stackoverflow.com/questions/54458815/pyspark-serialized-task-exceeds-max-allowed-consider-increasing-spark-rpc-mess

 

Pyspark: Serialized task exceeds max allowed. Consider increasing spark.rpc.message.maxSize or using broadcast variables for lar

I'm doing calculations on a cluster and at the end when I ask summary statistics on my Spark dataframe with df.describe().show() I get an error: Serialized task 15:0 was 137500581 bytes, which exce...

stackoverflow.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[kotlin] repeat(n) { ... }  (0) 2023.02.23
[spark] Service 'sparkDriver' failed after 16 retries (on a random free port)! 오류  (0) 2023.02.23
[python] mecab (형태소 분석기)  (0) 2023.02.23
[zeppelin] add interpreter  (0) 2023.02.17
[yarn] cost tool `migration-hadoop-to-emr-tco-simulator`  (0) 2023.02.16
댓글