[aws] emr spark config

[
  {
    "Classification": "spark-env",
    "Configurations": [
      {
        "Classification": "export",
        "Properties": {
          "JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
        }
      }
    ]
  },
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64",
      "spark.sql.broadcastTimeout": "3600",
      "spark.yarn.executor.memoryOverheadFactor": "0.1",
      "spark.default.parallelism": "200",
      "spark.yarn.am.memory": "2g",
      "spark.executor.extraJavaOptions": "-XX:+IgnoreUnrecognizedVMOptions"
    }
  },
  {
    "Classification": "hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
  },
  {
    "Classification": "spark-hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
  },
  {
    "Classification": "yarn-site",
    "Properties": {
      "yarn.nodemanager.pmem-check-enabled": "false",
      "yarn.nodemanager.vmem-check-enabled": "false",
      "yarn.nodemanager.vmem-pmem-ratio": "5",
      "yarn.nodemanager.resource.cpu-vcores": "4",
      "yarn.nodemanager.resource.memory-mb": "253952",
      "yarn.scheduler.maximum-allocation-vcores": "128",
      "yarn.scheduler.maximum-allocation-mb": "253952"
    }
  },
  {
    "Classification": "spark",
    "Properties": {
      "maximizeResourceAllocation": "true"
    }
  }
]

https://www.facebook.com/groups/sparkkoreauser/posts/1323386911056542/

Facebook에 로그인

https://docs.aws.amazon.com/ko_kr/emr/latest/ReleaseGuide/emr-spark-configure.html

Spark 구성 - Amazon EMR

https://stackoverflow.com/questions/54596569/tables-not-found-in-spark-sql-after-migrating-from-emr-to-aws-glue

Tables not found in Spark SQL after migrating from EMR to AWS Glue

https://stackoverflow.com/questions/38988941/running-yarn-with-spark-not-working-with-java-8/39456967#39456967

Running yarn with spark not working with Java 8