티스토리 뷰

공부

[pyspark] Can I add arguments to python code when I submit spark job?

승가비 2023. 4. 17. 20:21
728x90
import os
import sys

n = int(sys.argv[1])
a = 2
tables = []
for _ in range(n):
    tables.append(sys.argv[a])
    a += 1
print(tables)

https://stackoverflow.com/questions/32217160/can-i-add-arguments-to-python-code-when-i-submit-spark-job

 

Can I add arguments to python code when I submit spark job?

I'm trying to use spark-submit to execute my python code in spark cluster. Generally we run spark-submit with python code like below. # Run a Python application on a cluster ./bin/spark-submit \...

stackoverflow.com

 

728x90
저작자표시 비영리

'공부' 카테고리의 다른 글

[croniter] match  (0) 2023.04.21
[python] datetime to int `int(datetime.datetime.utcnow().timestamp())`  (0) 2023.04.21
[gradle] allow insecure protocols, android gradle  (0) 2023.04.17
[java] java -cp /location_of_jar/myjar.jar com.mypackage.myClass  (0) 2023.04.17
[airflow] helm chart  (0) 2023.04.17
댓글