티스토리 뷰

공부

[spark] 자주 쓰이는 config

승가비 2023. 5. 26. 21:38
728x90
spark.master
spark.driver.extraClassPath
spark.eventLog.dir
spark.sql.warehouse.dir
spark.yarn.appMasterEnv
spark.executor.memory
spark.driver.memory
spark.executor.cores
spark.driver.cores
spark.executor.memoryOverhead
spark.executor.defaultJavaOptions
spark.driver.defaultJavaOptions
spark.dynamicAllocation.enabled
spark.dynamicAllocation.shuffleTracking.enabled
spark.dynamicAllocation.initialExecutors
spark.dynamicAllocation.maxExecutors
spark.dynamicAllocation.minExecutors
spark.shuffle.service.enabled

https://spidyweb.tistory.com/331

 

[Spark] Spark 자주 쓰이는 Config 정리 driver,executor memory, dynamic allocation

properties 분류 설명 예시 spark.master Application Properties 클러스터 매니저(리소스 매니저 설정) local[*], yarn, k8s://HOST:PORT, mesos: //HOST:PORT spark.driver.extraClassPath Runtime Environment 드라이버의 클래스 경로에

spidyweb.tistory.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[airflow] [emr_add_steps_operator] Allow job_flow_id & job_flow_name (Priority is in the order of id, name.)  (0) 2023.05.26
[maven] scope  (0) 2023.05.26
[slack] link <http://someurl|like this>  (0) 2023.05.26
[openssl] 1.1, 3.0, 3  (0) 2023.05.26
[github.actions] (node:1738) UnhandledPromiseRejectionWarning: HttpError: Resource not accessible by integration  (0) 2023.05.26
댓글