티스토리 뷰

공부

[Spark] `foreach` vs `foreachPartition`

승가비 2020. 12. 10. 10:29
728x90

https://stackoverflow.com/questions/30484701/apache-spark-foreach-vs-foreachpartitions-when-to-use-what

 

Apache Spark - foreach Vs foreachPartitions When to use What?

I would like to know if the foreachPartitions will results in better performance, due to an higher level of parallelism, compared to the foreach method considering the case in which I'm flowing th...

stackoverflow.com

https://stackoverflow.com/questions/49146426/does-sparks-foreachpartition-run-on-the-driver-or-on-the-worker

 

Does sparks foreachPartition run on the driver or on the worker?

It is unclear from documentation where exactly does the lambda given to the foreachpartition run- on the driver or on the worker?

stackoverflow.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[Python] beautify JSON  (0) 2020.12.10
[Spark] string to date  (0) 2020.12.10
[Sh] HDFS exists directory  (0) 2020.12.10
[Spark] Encounter SparkException “Cannot broadcast the table that is larger than 8GB”  (0) 2020.12.10
[Spring] security actuator disable  (0) 2020.11.28
댓글