공부

[Spark] Encounter SparkException “Cannot broadcast the table that is larger than 8GB”

승가비 2020. 12. 10. 10:19
728x90
'spark.sql.autoBroadcastJoinThreshold': '-1'

https://stackoverflow.com/questions/49567066/encounter-sparkexception-cannot-broadcast-the-table-that-is-larger-than-8gb

 

Encounter SparkException "Cannot broadcast the table that is larger than 8GB"

I am using Spark 2.2.0 to do data processing. I am using Dataframe.join to join 2 dataframes together, however I encountered this stack trace: 18/03/29 11:27:06 INFO YarnAllocator: Driver requ...

stackoverflow.com

 

728x90
저작자표시 비영리 (새창열림)