배우기를 멈추는 사람은 20세건 80세건 늙은 것이다.

Spark – Slow Load Into Partitioned Hive Table on S3 – Direct Writes, Output Committer Algorithms – Large-Scale Data Engine

I have a Spark job that transforms incoming data from compressed text files into Parquet format and loads them into a daily partition of a Hive table. This is a typical job in a data lake, it is quite simple but in my case it was very slow. Initially it to

cloudsqale.com

Small files in Hadoop

Problem

medium.com