티스토리 뷰

공부

[Spark] Total size of serialized results of 16 tasks bigger than spark.driver.maxResultSize

승가비 2020. 11. 7. 14:31
728x90
--conf spark.driver.maxResultSize=4g

https://stackoverflow.com/questions/47996396/total-size-of-serialized-results-of-16-tasks-1048-5-mb-is-bigger-than-spark-dr

 

Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

I get the following error when I add --conf spark.driver.maxResultSize=2050 to my spark-submit command. 17/12/27 18:33:19 ERROR TransportResponseHandler: Still have 1 requests outstanding when

stackoverflow.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[Python] inf 무한대  (0) 2020.11.07
[DP] 동전문제  (0) 2020.11.07
[MySQL] count connection  (0) 2020.11.07
[Pyspark] functions  (0) 2020.11.07
[Link] regexr.com  (0) 2020.11.07
댓글