티스토리 뷰

728x90
--conf spark.rpc.message.maxSize=2047

https://stackoverflow.com/questions/54458815/pyspark-serialized-task-exceeds-max-allowed-consider-increasing-spark-rpc-mess

 

Pyspark: Serialized task exceeds max allowed. Consider increasing spark.rpc.message.maxSize or using broadcast variables for lar

I'm doing calculations on a cluster and at the end when I ask summary statistics on my Spark dataframe with df.describe().show() I get an error: Serialized task 15:0 was 137500581 bytes, which exce...

stackoverflow.com

 

728x90
댓글