배우기를 멈추는 사람은 20세건 80세건 늙은 것이다.

spark.sql("refresh TABLE schema.table") https://stackoverflow.com/questions/73294636/spark-refresh-delta-table-in-s3 Spark: refresh Delta Table in S3 how can I run the refresh table command on a Delta Table in S3? When I do deltatable = DeltaTable.forPath(spark, "s3a://test-bucket/delta_table/") spark.catalog.refreshTable(deltatable) ... stackoverflow.com https://stackoverflow.com/questions/4923..

deactivate https://stackoverflow.com/questions/990754/how-to-leave-exit-deactivate-a-python-virtualenv How to leave/exit/deactivate a Python virtualenv I'm using virtualenv and the virtualenvwrapper. I can switch between virtualenv's just fine using the workon command. me@mymachine:~$ workon env1 (env1)me@mymachine:~$ workon env2 (env2)me@mymachin... stackoverflow.com

https://ovso.github.io/blog/2018/04/18/gradle-dependencies-exclude/ Gradle dependencies exclude(Gradle 의존성 제거) Gradle 의존성 제거 ovso.github.io

SELECT DATE_FORMAT(event_time, "yyyy-MM-dd'T'HH:mm:ss'Z'") event_time https://stackoverflow.com/questions/66679640/convert-timestamp-format-to-iso-time-format-in-pyspark convert timestamp format to iso time format in pyspark I have a pyspark dataframe that has a field, time, that has timestamps in two formats, "11-04-2019,00:32:13" and "2019-12-05T07:57:16.000Z" How can I convert all the timesta..

https://airflow.apache.org/docs/apache-airflow/1.10.13/_api/airflow/operators/index.html airflow.operators — Airflow Documentation airflow.apache.org

https://www.jenkins.io/doc/book/pipeline/syntax/#parameters Pipeline Syntax Scripted Pipeline, like Declarative Pipeline, is built on top of the underlying Pipeline sub-system. Unlike Declarative, Scripted Pipeline is effectively a general-purpose DSL built with Groovy. Most functionality provided by the Groovy language is made av www.jenkins.io https://waspro.tistory.com/554 [Jenkins] Pipeline ..

https://techblog.woowahan.com/2646/ 좌충우돌 Terraform 입문기 | 우아한형제들 기술블로그 {{item.name}} 안녕하세요, 저는 신사업부문의 오지산이라고 합니다. 저희 신사업부문은 배달의민족과는 조금 다른… 시장의 문제를 해결하기 위해 다양한 신규 서비스들이 개발되는 곳입니다. 그 techblog.woowahan.com https://velog.io/@khyup0629/Terraform-CloudTrail-%EC%9D%B4%EB%B2%A4%ED%8A%B8-%EB%A1%9C%EA%B7%B8%EB%A5%BC-CloudWatch-Logs%EB%A1%9C-%EB%AA%A8%EB%8B%88%ED%84%B0%EB%A7%81%ED%95%98%EB%8A%94-%EC%9D%B8%E..

mongosh use admin db.createUser({ user: "user" , pwd: "1234", roles: [ "readWrite", "dbAdmin" ] }) use metadata db.createCollection("product") # mongodb://user:1234@localhost:27017/metadata https://stackoverflow.com/questions/22418052/connect-to-a-specific-database-by-default-in-mongodb/74636800#74636800 Connect to a specific database by default in mongodb I am running a mongodb on a linux box. ..

https://stackoverflow.com/questions/32788322/how-to-add-a-constant-column-in-a-spark-dataframe How to add a constant column in a Spark DataFrame? I want to add a column in a DataFrame with some arbitrary value (that is the same for each row). I get an error when I use withColumn as follows: dt.withColumn('new_column', 10).head(5) ---------... stackoverflow.com

def my_func(a, *args, **kwargs): saved_args = locals() print("saved_args is", saved_args) local_var = 10 print("saved_args is", saved_args) print("But locals() is now", locals()) my_func(20, 30, 40, 50, kwarg1='spam', kwarg2='eggs') saved_args is {'a': 20, 'args': (30, 40, 50), 'kwargs': {'kwarg1': u'spam', 'kwarg2': u'eggs'}} saved_args is {'a': 20, 'args': (30, 40, 50), 'kwargs': {'kwarg1': u'..

import inspect arg_keys = inspect.getfullargspec({method})[0] >>> def foo(a, b, c=4, *arglist, **keywords): pass >>> inspect.getfullargspec(foo) (['a', 'b', 'c'], 'arglist', 'keywords', (4,)) https://stackoverflow.com/questions/218616/how-to-get-method-parameter-names How to get method parameter names? Given the Python function: def a_method(arg1, arg2): pass How can I extract the number and nam..

// package.json { "type": "module" } https://stackoverflow.com/questions/58384179/syntaxerror-cannot-use-import-statement-outside-a-module SyntaxError: Cannot use import statement outside a module I've got an ApolloServer project that's giving me trouble, so I thought I might update it and ran into issues when using the latest Babel. My "index.js" is: require('dotenv').config() imp... stackoverf..

https://www.guru99.com/data-lake-vs-data-warehouse.html Data Lake vs Data Warehouse – Difference Between Them What is Data Warehouse? A data warehouse is a blend of technologies and components which allows the strategic use of data. It is a technique for collecting and managing data from varied sources to pro www.guru99.com https://www.talend.com/resources/data-lake-vs-data-warehouse/ Data Lake ..

https://dailyheumsi.tistory.com/265 Feast - Quick Review 일반적인 정형 데이터 머신러닝 코드에는 데이터를 불러오고 필요한 feature를 뽑아 가공하는 부분이 있다. 보통 데이터 웨어하우스나 아니면 원천 데이터 소스에서 데이터를 불러올텐데, 이렇게 dailyheumsi.tistory.com https://docs.feast.dev/ Introduction - Feast Decouple ML from data infrastructure by providing a single data access layer that abstracts feature storage from feature retrieval, ensuring models remain portable a..

Generation Usage Description First – s3 s3:\\ s3 which is also called classic (s3: filesystem for reading from or storing objects in Amazon S3 This has been deprecated and recommends using either the second or third generation library. Second – s3n s3n:\\ s3n uses native s3 object and makes easy to use it with Hadoop and other files systems. This is also not the recommended option. Third – s3a s..

"--conf", "spark.yarn.appMasterEnv.JAVA_HOME=/usr/lib/jvm/java-11-amazon-corretto.x86_64", https://brocess.tistory.com/176 [ Spark ] 스파크 jdk버전 바꿔서 실행하기 현상황 : Cloudera(클라우데라) 버전(CDH 5.5.1, Parcel), Spark버전(1.5) - jdk version 1.7필요상황 : 기존 작업을 Spark1.5(jdk1.7) - jdk 1.8로 돌리기준비상황 : 클러스터의 각 노드들에 jdk1.8이 설치되어 있어야 함. brocess.tistory.com

aws s3 ls s3://bucket --recursive | grep -v -E "(Bucket: |Prefix: |LastWriteTime|^$|--)" | awk 'BEGIN {total=0}{total+=$3}END{print total/1024/1024/1024" GB"}' https://gist.github.com/stefhen/06e9e87cb28eb46b9e34 aws cli s3 du aws cli s3 du. GitHub Gist: instantly share code, notes, and snippets. gist.github.com

https://www.gorillastack.com/blog/real-time-events/cloudtrail-vs-cloudwatch/ CloudTrail vs CloudWatch - A Detailed Guide - GorillaStack Learn the difference between AWS CloudTrail and CloudWatch and when to use them. Details on CloudTrail vs CloudWatch Events, Alarms, Metrics & Logs. www.gorillastack.com

https://weejw.tistory.com/200 Spark Streaming,Structured Streaming http://weejw.tistory.com/35 무려 작년 7월초 spark streaming을 공부해보자 라는 게시글에 .... 드디어 스트리밍 공부를 한다! ٩(ˊᗜˋ*)و Spark Streaming 스트리밍 데이터는 날씨나 log 데이터 처럼 계속해서 생 weejw.tistory.com

https://aws.amazon.com/ko/premiumsupport/knowledge-center/s3-empty-bucket-lifecycle-rule/ 수명 주기 구성 규칙을 사용하여 Amazon S3 버킷 비우기 닫기 Shaji 씨의 동영상을 통해 자세히 알아보기(4:03) aws.amazon.com https://docs.3rdeyesys.com/aws/aws_s3_lifecycle_management.html#%EC%A3%BC%EC%9D%98%EC%82%AC%ED%95%AD AWS S3 수명 주기 (LifeCycle) 설정하기 AWS S3 수명 주기 (LifeCycle) 설정하기 docs.3rdeyesys.com https://aws.amazon.com/ko/premiumsupport/kn..

https://gist.github.com/seunggabi/f8224d33b81dca80ded01b9a5888030c aws-cli-configure.sh aws-cli-configure.sh. GitHub Gist: instantly share code, notes, and snippets. gist.github.com brew install awscli #aws configure #AWS Access Key ID [None] : [발급받은 IAM의 Access Key ID] #AWS Secret Access Key [None] : [발급받은 IAM의 Secret Access Key] #Default region name [None] : ap-northeast-2[서울 리전] #Default outp..

AIRFLOW__CORE__LOAD_EXAMPLES: false https://stackoverflow.com/questions/43410836/how-to-remove-default-example-dags-in-airflow How to remove default example dags in airflow I am a new user of Airbnb's open source workflow/datapipeline software airflow. There are dozens of default example dags after the web UI is started. I tried many ways to remove these dags, but I've stackoverflow.com environm..

https://docs.aws.amazon.com/ko_kr/AmazonS3/latest/userguide/lifecycle-configuration-examples.html#lifecycle-config-conceptual-ex4 S3 수명 주기 구성의 예제 - Amazon Simple Storage Service 동일한 규칙에 Days 및 ExpiredObjectDeleteMarker 태그를 모두 지정할 수는 없습니다. Days 태그를 지정하면 삭제 마커의 기간이 사용 기간 기준을 충족할 때 Amazon S3가 자동으로 ExpiredObjectDeleteMarker 정리 docs.aws.amazon.com

https://hudi.apache.org/cn/blog/2022/01/14/change-data-capture-with-debezium-and-apache-hudi/ Change Data Capture with Debezium and Apache Hudi | Apache Hudi As of Hudi v0.10.0, we are excited to announce the availability of Debezium sources for Deltastreamer that provide the ingestion of change capture data (CDC) from Postgres and Mysql databases to your data lake. For more details, please refe..

HTTP POST default curl \ --data-urlencode "paramName=value" \ --data-urlencode "secondParam=value" \ http://example.com curl --get \ --data-urlencode "p1=value 1" \ --data-urlencode "p2=value 2" \ http://example.com # http://example.com?p1=value%201&p2=value%202 https://stackoverflow.com/questions/296536/how-to-urlencode-data-for-curl-command How to urlencode data for curl command? I am trying t..

https://artifacthub.io/packages/helm/airflow-helm/airflow airflow 8.6.1 · airflow-helm/airflow-helm Airflow Helm Chart (User Community) - the standard way to deploy Apache Airflow on Kubernetes with Helm artifacthub.io

https://www.lainyzine.com/ko/article/how-to-install-wsl2-and-use-linux-on-windows-10/ [Windows 10] WSL2 설치 및 사용법 Microsoft에서는 2020년 5월 리눅스를 윈도우와 통합해서 사용할 수 있는 WSL2를 발표했습니다. 이 글에서는 WSL2를 설치하고 사용하는 방법을 소개합니다 www.lainyzine.com

https://bcho.tistory.com/1327 SRE #2-SRE는 어떻게 일하는가? SRE는 어떻게 일하는가? 조대협 (http://bcho.tistory.com) 이글은 앞의 글 "SRE/DEOPS의 개념과 SRE는 무엇을 하는가?" (https://bcho.tistory.com/1325) 와 연결된 글입니다.How SRE does Devops?그럼 SRE들은 이런한 일들을 bcho.tistory.com https://bcho.tistory.com/1328 SRE #3-SRE의 주요 지표 SLI/SLO (Service Level Indicatior, Service Level Objectives) SRE #3-SRE 주요 지표 (SLI/SLO)조대협 (http://bcho.tistory.com)..

function requestUtils(method, url, body, f, bearer) { const xhr = new XMLHttpRequest(); xhr.open(method, url, true); xhr.withCredentials = true; xhr.setRequestHeader("Content-Type", "application/x-www-form-urlencoded"); bearer && xhr.setRequestHeader("Authorization", bearer); xhr.onreadystatechange = function () { if (this.readyState === XMLHttpRequest.DONE && this.status === 200) { f(this.respo..