티스토리 뷰

728x90
'spark.sql.autoBroadcastJoinThreshold': '-1'

https://stackoverflow.com/questions/49567066/encounter-sparkexception-cannot-broadcast-the-table-that-is-larger-than-8gb

 

Encounter SparkException "Cannot broadcast the table that is larger than 8GB"

I am using Spark 2.2.0 to do data processing. I am using Dataframe.join to join 2 dataframes together, however I encountered this stack trace: 18/03/29 11:27:06 INFO YarnAllocator: Driver requ...

stackoverflow.com

 

728x90

'공부' 카테고리의 다른 글

[Spark] `foreach` vs `foreachPartition`  (0) 2020.12.10
[Sh] HDFS exists directory  (0) 2020.12.10
[Spring] security actuator disable  (0) 2020.11.28
[Docker] docker attach ${name}  (0) 2020.11.28
[Kafka] describe & delete  (0) 2020.11.28
댓글