[spark] sql repartition (by query hint) `SELECT /*+ COALESCE(1) */` is better than `SELECT /*+ REPARTITION(1) */`

fun insertOverwrite(target: String, db: DB, table: Table, partition: String, columns: List<String>) {
    execute(
        """
        INSERT OVERWRITE TABLE ${db.name}.${table.name}
        PARTITION ($partition)
        SELECT /*+ REPARTITION(1) */ ${columns.concat(ConstUtil.COMMA)}
        FROM $target
        """.trimIndent()
    )
}

https://stackoverflow.com/questions/46932373/how-to-consolidate-results-of-a-spark-sql-query-to-avoid-lots-of-small-files-a

How to consolidate results of a spark SQL query to avoid lots of small files / avoid empty files

https://kontext.tech/article/1155/use-spark-sql-partitioning-hints

Use Spark SQL Partitioning Hints

https://github.com/dhkdn9192/data_engineer_should_know/blob/master/interview/hadoop/difference_between_repartition_and_coalesce_in_spark.md

GitHub - dhkdn9192/data_engineer_should_know: 데이터 엔지니어가 알아야 하는 것들