티스토리 뷰

공부

[Spark] hive dynamic partition

승가비 2020. 12. 13. 00:43
728x90
spark-shell \
  --conf "spark.hadoop.hive.exec.dynamic.partition=true" \
  --conf "spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict" \
  ...

https://stackoverflow.com/questions/58633753/ignoring-non-spark-config-property-hive-exec-dynamic-partition-mode

 

Ignoring non-spark config property: hive.exec.dynamic.partition.mode

How to run a Spark-shell with hive.exec.dynamic.partition.mode=nonstrict? I try (as suggested here) export SPARK_MAJOR_VERSION=2; spark-shell --conf "hive.exec.dynamic.partition.mode=nonstrict...

stackoverflow.com

https://stackoverflow.com/questions/31341498/save-spark-dataframe-as-dynamic-partitioned-table-in-hive

 

Save Spark dataframe as dynamic partitioned table in Hive

I have a sample application working to read from csv files into a dataframe. The dataframe can be stored to a Hive table in parquet format using the method df.saveAsTable(tablename,mode). The ab...

stackoverflow.com

https://forums.databricks.com/questions/8555/ive-set-the-partition-mode-to-nonstrict-in-hive-bu.html

 

I've set the partition mode to nonstrict in hive but spark is not seeing it - Databricks Community Forum

 

forums.databricks.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[Java] new ArrayList<>(Arrays.asList("a", "b", "c"))  (0) 2020.12.13
[Spark] hadoop.fs.default.name (default file system)  (0) 2020.12.13
[Python] dict to string  (0) 2020.12.10
[Python] call method by name of function  (0) 2020.12.10
[Python] beautify JSON  (0) 2020.12.10
댓글