공부

[Spark] Total size of serialized results of 16 tasks bigger than spark.driver.maxResultSize

승가비 2020. 11. 7. 14:31
728x90
--conf spark.driver.maxResultSize=4g

https://stackoverflow.com/questions/47996396/total-size-of-serialized-results-of-16-tasks-1048-5-mb-is-bigger-than-spark-dr

 

Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

I get the following error when I add --conf spark.driver.maxResultSize=2050 to my spark-submit command. 17/12/27 18:33:19 ERROR TransportResponseHandler: Still have 1 requests outstanding when

stackoverflow.com

 

728x90
저작자표시 비영리 (새창열림)