배우기를 멈추는 사람은 20세건 80세건 늙은 것이다.

https://docs.cloudera.com/documentation/enterprise/5-9-x/topics/admin_hbase_filtering.html HBase Filtering | 5.9.x | Cloudera Documentation When reading data from HBase using Get or Scan operations, you can use custom filters to return a subset of results to the client. While this does not reduce server-side IO, it does reduce network bandwidth and reduces the amount of data the client needs t d..

last week, I had a traffic issue in kafka. Traffic of kafka has increase tenfold. I increased partition size.(6 -> 12 -> 20), but 20 partitions is not enough. I can not increase more than 20, beacause this traffic is temporal. (This traffic lasted 5 days.) When traffic is recovers, I put annother 8 kafka-consumer in a day. My kafka lag message of consumer recovered. but, partition skewed. I do `..

https://www.youtube.com/watch?v=_w7Ksdh_wxI

https://wiki.jenkins.io/display/JENKINS/Standard+Security+Setup Standard Security Setup - Jenkins - Jenkins Wiki Immediately after installation, Jenkins will allow anyone to run anything as user jenkins, which is bad. This page shows you how to set up basic security using the Configure Global Security page. The Configure Global Security page has two sections i wiki.jenkins.io

### broker: 5, partition: 6, 1k/s (ok) ### broker: 5, partition: 20, 100k/s (not ok) - partition ㄴ more than 100 ㄴ broker * n (because skewed) https://sori-nori.gitlab.io/docs/how-to-choose-the-number-of-partition-in-a-kafka-cluster/ Kafka Topic의 partition 수 정하기 Kafka Topic의 partition 수를 정하는 방법 sori-nori.gitlab.io

RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory()); https://blog.voidmainvoid.net/224 Resttemplate으로 PATCH 사용시 에러나는 경우 해결방법 - Invalid HTTP method: PATCH; nested exception is java.net. Spring boot 내에서 Resttemplate를 사용하여 타 url을 호출하는 경우가 있다. 사용하던 도중 아래와 같은 오류가 날 때가 있다. RestTemplate restTemplate = new RestTemplate(); HttpHeaders headers = new Ht.. blog.voidmai..

@RequestMapping(value = "/user?${id}", method = RequestMethod.GET) public ResponseEntity getUser(@PathVariable Long id) { User user = ...; if (user != null) { return new ResponseEntity(user, HttpStatus.OK); } return new ResponseEntity(HttpStatus.NOT_FOUND); } https://stackoverrun.com/ko/q/6166428 spring - 스프링 컨트롤러에서 상태를 찾을 수 없음을 반환하는 방법 다음 스프링 컨트롤러 코드가 있으며 데이터베이스에 사용자가없는 경우 상태를 반환하지 않으려면 어떻게해야합니..

http://mwultong.blogspot.com/2007/09/c-0-printf-align.html C언어, 숫자 끝에 0 붙이기, 실수 앞뒤 영 넣어 줄맞추기; printf Align C에서 실수 숫자를 출력할 때, 숫자 맨 뒤에 0을 붙이거나, 숫자 앞뒤에 모두 0(제로)을 붙여서 줄을 맞추어 출력하는 방법입니다. printf 함수에 "폭"을 지정해 주면 됩니다. 다음 예제와 같습니다. mwultong.blogspot.com

s=$(date +%s) sleep 3s e=$(date +%s) diff=`expr ${e} - ${s}` if [ ${diff} -lt 30 ]; then echo $diff exit 1 fi https://qastack.kr/ubuntu/892604/meaning-of-exit-0-exit-1-and-exit-2-in-a-bash-script bash 스크립트에서 종료 0, 종료 1 및 종료 2의 의미 qastack.kr https://stackoverflow.com/questions/7119130/less-than-operator-in-if-statement-results-in-no-such-file-or-directory Less than operator '

https://gmlwjd9405.github.io/2018/12/25/difference-dao-dto-entity.html [DAO] DAO, DTO, Entity Class의 차이 - Heee's Development Blog Step by step goes a long way. gmlwjd9405.github.io DAO: 로직 DTO: beans

https://knight76.tistory.com/entry/hive [hive] 정렬 키워드 - order by, sort by, cluster by, distribute by https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy 하이브에서 사용되는 정렬 키워드를 소개한다. * ORDER BY (ASC|DESC): RDMBS의 ORDER BY 문과 비슷하다. ORDER BY 문을 실행 시, 하나.. knight76.tistory.com

https://monkey3199.github.io/develop/python/2018/12/04/python-pararrel.html Nathan's Blog The blog to learn more. monkey3199.github.io

df .repartition(10) // No. of concurrent connection Spark to PostgreSQL .write.format('jdbc').options( url=psql_url_spark, driver=spark_env['PSQL_DRIVER'], dbtable="{schema}.{table}".format(schema=schema, table=table), user=spark_env['PSQL_USER'], password=spark_env['PSQL_PASS'], batchsize=2000000, queryTimeout=690 ).mode(mode).save() https://stackoverflow.com/questions/58676909/how-to-speed-up-..

spark = SparkSession .builder .appName("Your App") .config("spark.sql.broadcastTimeout", "36000") .getOrCreate() https://stackoverflow.com/questions/41123846/why-does-join-fail-with-java-util-concurrent-timeoutexception-futures-timed-ou Why does join fail with "java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]"? I am using Spark 1.5. I have two dataframes of the form: ..

https://stackoverflow.com/questions/43415974/sort-array-order-by-a-different-column-hive sort_array order by a different column, Hive I have two columns, one of products, and one of the dates they were bought. I am able to order the dates by applying the sort_array(dates) function, but I want to be able to sort_array(products) by... stackoverflow.com

https://reference-m1.tistory.com/211 배포 전략의 종류(롤링/카나리 배포/블루 그린) 요즘은 MSA 아키텍처를 많이 지향하고 있는 추세이다. 이런 트렌드에 맞춰 배포 전략도 다양하게 개발되고 발전하여 변화하고 있다. 1. 롤링(Rolling) 일반적인 배포를 의미하며, 단순하게 서버를 reference-m1.tistory.com

spark-submit \ --master spark://Spark master_url \ -–conf spark.yarn.keytab=path_to_keytab \ -–conf spark.yarn.principal=principal@REALM.COM \ --class main-class application-jar hdfs://namenode:9000/path/to/input https://www.ibm.com/support/knowledgecenter/SSZU2E_2.3.0/managing_cluster/kerberos_hdfs_keytab.html

http://dveamer.github.io/backend/InitializingJavaVariable.html Dveamer 현실에서 살고 있지만 이상에 대한 꿈을 버리지 못한 몽상가의 홈페이지 입니다. 개인적인 기록을 주 목적으로 하며 일상과 프로그래밍 관련 글을 포스팅합니다. dveamer.github.io

SparkContext context = new SparkContext(new SparkConf().setAppName("spark-ml").setMaster("local[*]") .set("spark.hadoop.fs.default.name", "hdfs://localhost:54310").set("spark.hadoop.fs.defaultFS", "hdfs://localhost:54310") .set("spark.hadoop.fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()) .set("spark.hadoop.fs.hdfs.server", org.apache.hadoop.hdfs.server.namenode.Name..

spark-shell \ --conf "spark.hadoop.hive.exec.dynamic.partition=true" \ --conf "spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict" \ ... https://stackoverflow.com/questions/58633753/ignoring-non-spark-config-property-hive-exec-dynamic-partition-mode