배우기를 멈추는 사람은 20세건 80세건 늙은 것이다.

https://stackoverflow.com/questions/12464636/how-to-set-variables-in-hive-scripts How to set variables in HIVE scripts I'm looking for the SQL equivalent of SET varname = value in Hive QL I know I can do something like this: SET CURRENT_DATE = '2012-09-16'; SELECT * FROM foo WHERE day >= @CURRENT_DATE But th... stackoverflow.com

https://api.slack.com/reference/surfaces/formatting Formatting text for app surfaces A guide to using the mrkdwn syntax for your app-published text in messages, modals, and tabs. api.slack.com

import tableauserverclient as TSC SITE_LUID = "..." auth = TSC.PersonalAccessTokenAuth( TABLEAU["TOKEN_NAME"], TABLEAU["TOKEN_VALUE"], ) server = TSC.Server( TABLEAU["SERVER_URL"], use_server_version=True ) server.auth.sign_in(auth) for w in TSC.Pager(server.workbooks): server.workbooks.populate_connections(w) for c in w.connections: if c.datasource_id == SITE_LUID: server.workbooks.populate_vie..

https://stackoverflow.com/questions/35789412/spark-sql-difference-between-gzip-vs-snappy-vs-lzo-compression-formats Spark SQL - difference between gzip vs snappy vs lzo compression formats I am trying to use Spark SQL to write parquet file. By default Spark SQL supports gzip, but it also supports other compression formats like snappy and lzo. What is the difference between these stackoverflow.co..

https://dojang.io/mod/page/view.php?id=2400 파이썬 코딩 도장: 38.3 예외 발생시키기 지금까지 숫자를 0으로 나눴을 때 에러, 리스트의 범위를 벗어난 인덱스에 접근했을 때 에러 등 파이썬에서 정해진 예외만 처리했습니다. 이번에는 우리가 직접 예외를 발생시켜 보겠습니다. 예 dojang.io raise BadRequest(f"[Error] name: {name}; not found")

https://www.bangseongbeom.com/sys-path-pythonpath.html sys.path, PYTHONPATH: 파이썬 파일 탐색 경로 import 문을 통해 다른 파이썬 파일을 불러올 때, 파이썬은 내부적으로 파일을 찾기 위해 sys.path와 PYTHONPATH에 있는 경로를 탐색합니다. 이 두 변수를 적절히 수정해 임의의 디렉터리에 있는 파이썬 www.bangseongbeom.com

sudo systemctl stop zeppelin sudo systemctl status zeppelin sudo systemctl start zeppelin sudo systemctl status zeppelin https://aws.amazon.com/ko/premiumsupport/knowledge-center/restart-service-emr/ Amazon EMR에서 서비스 다시 시작하기 닫기 Sindhuri 씨의 동영상을 보고 자세히 알아보기(3:35) aws.amazon.com

from config import TABLEAU import tableauserverclient as TSC from util import timestamp auth = TSC.PersonalAccessTokenAuth( TABLEAU["TOKEN_NAME"], TABLEAU["TOKEN_VALUE"], ) server = TSC.Server( TABLEAU["SERVER_URL"], use_server_version=True ) target = "asdf" with server.auth.sign_in(auth): for v in TSC.Pager(server.views): if target == v.name: view = v break server.views.populate_pdf(view, TSC.I..

https://www.linkedin.com/pulse/orc-vs-parquet-vivek-singh/ ORC vs Parquet ORC and Parquet are both columnar formats and there has been a lot of debate on which performs better in terms of compression and performance. Dataset used in this benchmark is a publicly available dataset. www.linkedin.com https://medium.com/@dhareshwarganesh/benchmarking-parquet-vs-orc-d52c39849aef Benchmarking PARQUET v..

SELECT * FROM db_name."table_name$partitions" ORDER BY column_name DESC https://docs.aws.amazon.com/ko_kr/athena/latest/ug/show-partitions.html SHOW PARTITIONS - Amazon Athena 이 페이지에 작업이 필요하다는 점을 알려 주셔서 감사합니다. 실망시켜 드려 죄송합니다. 잠깐 시간을 내어 설명서를 향상시킬 수 있는 방법에 대해 말씀해 주십시오. docs.aws.amazon.com https://github.com/awsdocs/amazon-athena-user-guide/pull/89 (#88) feat: add db_name by seunggabi · Pull Request..

variables = ['first', 'second', 'third'] def run_dag_task(variable): task = dag_task(variable) return task task_arr=[] task_arr.append(run_dag_task(variable[0])) for variable in variables[1:]: task=run_dag_task(variable) task_arr[-1]>>task task_arr.append(task) https://stackoverflow.com/questions/70002086/how-to-run-tasks-sequentially-in-a-loop-in-an-airflow-dag How to run tasks sequentially in ..

https://stackoverflow.com/questions/36747268/why-does-conf-setspark-app-name-appname-not-set-the-name-in-the-ui Why does conf.set("spark.app.name", appName) not set the name in the UI? I am calling val appName : String = arguments.getNameFromConfig val conf = new SparkConf() conf.set("spark.driver.maxResultSize", "30G") conf.set("spark.app.name", appName) println("Master: " + stackoverflow.com

grep -v 'exclude_word' file egrep -v '(main|master)' file https://stackoverflow.com/questions/4538253/how-can-i-exclude-one-word-with-grep How can I exclude one word with grep? I need something like: grep ^"unwanted_word"XXXXXXXX stackoverflow.com https://www.warp.dev/terminus/grep-exclude How To Exclude Patterns or Files With Grep [#excluding-single-pattern]Excluding a single pattern[#excluding..

https://spark.apache.org/docs/latest/sql-data-sources-text.html Text Files - Spark 3.3.1 Documentation spark.apache.org

PARTITIONED BY (dt string) CLUSTERED BY (user_key) SORTED BY (user_key ASC) INTO 256 BUCKETS CLUSTERED BY ~ SORTED BY ~ INTO {size} BUCKETS 을 사용해도, spark sql plan partitioning 작업에는 영향 없음. 비용이 많이 나온 것과 관련해서는, 로드되는 data size가 커서 발생하는 것 같음. 향후에, small files merge 를 통해서, 비용을 최적화할 수 있음. https://sparkbyexamples.com/apache-hive/hive-partitioning-vs-bucketing-with-examples/ Hive Partitioning vs Bucketin..

scp -i keypair-asdf.pem -r hadoop@asdf:~/tez.tar.gz . scp -i keypair-qwer.pem -r tez.tar.gz hadoop@qwer-emr:~/tez.tar.gz https://doheejin.github.io/linux/2021/03/03/linux-scp.html [Linux] scp 명령어로 (로컬↔서버) 파일 전송 scp는 SecureCopy의 약자로, 원격서버에 있는 파일과 폴더를 전송하거나 가져오기 위해 사용하는 명령어이다.ssh 원격 접속 프로토콜을 기반으로 하며, ssh와 동일한 22번 포트를 이용하기 때문에 passw doheejin.github.io

https://stackoverflow.com/questions/51933568/how-to-retrieve-hive-table-partition-location How to retrieve Hive table Partition Location? Show Partitions --> In Hive/Spark, this command only provides the Partition, without providing the location information on hdfs/s3 Since we maintain different location for each partition in a tab... stackoverflow.com

https://www.projectpro.io/recipes/explain-study-of-spark-query-execution-plans-using-explain Explain Study of Spark query execution plans using explain() - This recipe explains Study of Spark query execution plans using explain() www.projectpro.io

aws s3 sync . s3://asdf/a/b/c/ --delete​ aws s3 sync s3://my-bucket s3://my-other-bucket \ --exclude 'customers/*' \ --exclude 'orders/*' \ --exclude 'reportTemplate/*' https://stackoverflow.com/questions/32393026/exclude-multiple-folders-using-aws-s3-sync Exclude multiple folders using AWS S3 sync How to exclude multiple folders while using aws s3 syn ? I tried : # aws s3 sync s3://inksedge-app..