티스토리 뷰

공부

[zeppelin] spark.yarn.stagingDir (x) -> spark.hadoop.hive.exec.stagingdir (o)

승가비 2023. 1. 21. 03:18
728x90
spark.hadoop.hive.exec.stagingdir: s3://temp/.hive-staging

https://vanducng.dev/2020/12/05/Compact-multiple-small-files-on-HDFS/

 

Compact multiple small files on HDFS

Hadoop can handle with very big file size, but will encounter performance issue with too many files with small size. The reason is explained in detailed from here. In short, every single on a data n

vanducng.dev

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[s3] `_$folder$` 접미사가 있는 빈 파일을 안전하게 삭제할 수 있습니까?  (0) 2023.01.24
[s3] bucket naming  (0) 2023.01.24
[airflow] RBAC  (0) 2023.01.17
[chrome] tabs.executeScript()  (0) 2023.01.17
[bitbucket] git clone in pipeline  (0) 2023.01.17
댓글