[scala] loop & data = data ++ data
val path = "s3://data/seunggabi/orc"
val extension = path.substring(path.lastIndexOf("/") + 1)
var data = Seq(
("asdf", "asdf")
)
for(a <- 1 to 13) {
data = data ++ data
}
for(a <- 1 to 100) {
val df = spark.createDataFrame(data).toDF("a", "b")
df
.repartition(1)
.write
.format(extension)
.mode(SaveMode.Append)
.save(path)
}https://www.tutorialspoint.com/scala/scala_for_loop.htm
Scala - for Loops
Scala for Loops - A for loop is a repetition control structure that allows you to efficiently write a loop that needs to execute a specific number of times. There are various forms of for loop in Scala which are described below −
www.tutorialspoint.com
https://alvinalexander.com/scala/how-to-append-prepend-items-vector-seq-in-scala/
Scala: How to append and prepend items to Vector and Seq | alvinalexander.com
Scala FAQ: How do I append or prepend one or more elements to a Vector or Seq class? Solution To append or prepend one or more elements to a Vector or Seq, use these methods: to append one item, use :+ to append multiple items, use ++ to prepend one item,
alvinalexander.com
https://stackoverflow.com/questions/27033823/how-to-overwrite-the-output-directory-in-spark
How to overwrite the output directory in spark
I have a spark streaming application which produces a dataset for every minute. I need to save/overwrite the results of the processed data. When I tried to overwrite the dataset org.apache.hadoop....
stackoverflow.com
Solved: Unable to instantiate SparkSession with Hive support because Hive classes are not found - Spark By {Examples}
While running a Spark application with Hive enabled getting the below error message? Exception in thread "main" java.lang.IllegalArgumentException: Unable
sparkbyexamples.com