티스토리 뷰

공부

[zeppelin] org.apache.spark.sql.execution.datasources.FileDownloadException: Failed to download file path: s3://bucket/prefix/partition.snappy.parquet, range: 0-16165503, partition values: [empty row], isDataPresent: false, eTag: 53ea26b5ecc9a194efe5163..

승가비 2022. 12. 6. 13:38
728x90
spark.sql("refresh TABLE schema.table")

https://stackoverflow.com/questions/73294636/spark-refresh-delta-table-in-s3

 

Spark: refresh Delta Table in S3

how can I run the refresh table command on a Delta Table in S3? When I do deltatable = DeltaTable.forPath(spark, "s3a://test-bucket/delta_table/") spark.catalog.refreshTable(deltatable) ...

stackoverflow.com

https://stackoverflow.com/questions/49234471/when-to-execute-refresh-table-my-table-in-spark

 

When to execute REFRESH TABLE my_table in spark?

Consider a code; import org.apache.spark.sql.hive.orc._ import org.apache.spark.sql._ val path = ... val dataFrame:DataFramew = ... val hiveContext = new org.apache.spark.sql.hive.HiveContext(

stackoverflow.com

https://stackoverflow.com/questions/72732739/cache-error-when-querying-table-stored-in-glue-data-catalog-through-zeppelin/74697331#74697331

 

Cache error when querying table stored in Glue Data Catalog through Zeppelin

I have an error with the way Zeppelin cache tables. We update the data in the Glue Data Catalog in real time, so when we want to query a partition that was updated using Spark, sometimes we get the

stackoverflow.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[spark] `spark.yarn.stagingDir`  (0) 2022.12.06
[python] jinja2 + f-string  (0) 2022.12.06
[python] venv deactivate  (0) 2022.12.03
[gradle] exclude dependency  (0) 2022.12.02
[SQL] `ISO8601` DATE_FORMAT(event_time, "yyyy-MM-dd'T'HH:mm:ss'Z'") event_time,  (0) 2022.12.02
댓글