티스토리 뷰

공부

[Spark] Encounter SparkException “Cannot broadcast the table that is larger than 8GB”

승가비 2020. 12. 10. 10:19
728x90
'spark.sql.autoBroadcastJoinThreshold': '-1'

https://stackoverflow.com/questions/49567066/encounter-sparkexception-cannot-broadcast-the-table-that-is-larger-than-8gb

 

Encounter SparkException "Cannot broadcast the table that is larger than 8GB"

I am using Spark 2.2.0 to do data processing. I am using Dataframe.join to join 2 dataframes together, however I encountered this stack trace: 18/03/29 11:27:06 INFO YarnAllocator: Driver requ...

stackoverflow.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[Spark] `foreach` vs `foreachPartition`  (0) 2020.12.10
[Sh] HDFS exists directory  (0) 2020.12.10
[Spring] security actuator disable  (0) 2020.11.28
[Docker] docker attach ${name}  (0) 2020.11.28
[Kafka] describe & delete  (0) 2020.11.28
댓글