티스토리 뷰

728x90
import org.apache.spark.sql.functions.input_file_name

df.withColumn("filename", input_file_name())

https://stackoverflow.com/questions/39868263/spark-load-data-and-add-filename-as-dataframe-column

 

Spark load data and add filename as dataframe column

I am loading some data into Spark with a wrapper function: def load_data( filename ): df = sqlContext.read.format("com.databricks.spark.csv")\ .option("delimiter", "\t")\ .opti...

stackoverflow.com

https://docs.databricks.com/sql/language-manual/functions/input_file_name.html

 

input_file_name function (Databricks SQL) | Databricks on AWS

 

docs.databricks.com

 

728x90
댓글
댓글쓰기 폼