[spark] broadcast nested loop join

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
sql("select * from table_withNull where id not in (select id from tblA_NoNull)").explain(true)

not exists를 사용하면 쿼리가 SortMergeJoin과 함께 실행됩니다.

 

https://www.bigdatainrealworld.com/how-does-broadcast-nested-loop-join-work-in-spark/

How does Broadcast Nested Loop Join work in Spark?

https://learn.microsoft.com/ko-kr/azure/databricks/kb/sql/disable-broadcast-when-broadcastnestedloopjoin

쿼리 계획에 BroadcastNestedLoopJoin이 있는 경우 브로드캐스트 사용하지 않도록 설정 - Azure Databricks