공부
[pyspark] Can I add arguments to python code when I submit spark job?
승가비
2023. 4. 17. 20:21
728x90
import os
import sys
n = int(sys.argv[1])
a = 2
tables = []
for _ in range(n):
tables.append(sys.argv[a])
a += 1
print(tables)Can I add arguments to python code when I submit spark job?
I'm trying to use spark-submit to execute my python code in spark cluster. Generally we run spark-submit with python code like below. # Run a Python application on a cluster ./bin/spark-submit \...
stackoverflow.com
728x90