티스토리 뷰

공부

[spark] org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of XXXX tasks (X.0 GB) is bigger than spark.driver.maxResultSize (X.0 GB)

승가비 2022. 12. 20. 22:27
728x90
  --conf spark.driver.maxResultSize=4g

https://wooono.tistory.com/41

 

[Spark] spark.driver.maxResultSize 오류

오류 org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of XXXX tasks (X.0 GB) is bigger than spark.driver.maxResultSize (X.0 GB) 원인 RDD로 분산 돼 있던 데이터를 collect() 등을 사용해 dri

wooono.tistory.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[spark] explain(true), explain("cost")  (0) 2022.12.20
[aws] aws s3 sync --exclude & --delete  (0) 2022.12.20
[aws] aws s3 cp s3://myBucket/dir localdir --recursive  (0) 2022.12.20
[hive] `--hiveconf hive.session.id=appName` & `--hiveconf hive.query.bane=appName`  (0) 2022.12.20
[hdfs] fsck  (0) 2022.12.20
댓글