공부
[spark] org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of XXXX tasks (X.0 GB) is bigger than spark.driver.maxResultSize (X.0 GB)
승가비
2022. 12. 20. 22:27
728x90
--conf spark.driver.maxResultSize=4g
[Spark] spark.driver.maxResultSize 오류
오류 org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of XXXX tasks (X.0 GB) is bigger than spark.driver.maxResultSize (X.0 GB) 원인 RDD로 분산 돼 있던 데이터를 collect() 등을 사용해 dri
wooono.tistory.com
728x90