AWS EKS airflow EC2 S3 SQL Python kubernetes lambda kafka Terraform MQTT MSK RDS IOTCore ecr kinesis CloudWatch FLUENTD docker mariaDB Jenkins GIT VPC Java glue catalog 견고한 데이터 엔지니어링 견고한 데이터엔지니어링 Victoria Metrics flink Best Practice Dynamic Partitioning Fluent Bit Managed Nodegroup superset Kinesis Firehose docker compose Apache Kafka OpenSearch Github IAC logging iceberg Transaction CI CD MySQL dynamic block jinja template user permission duckdb aws wrangler python env custom schedule lakehouse pyiceberg amazon quiksight airflow xcom serverless function http trigger function google-cloud-storage 데이터엔지니어링이란 견데엔 KubernetsExecutor git-sync celery executor teraform state local state kubernetes metric dag_run configuration dagRuns Dag trigger branch 관리 DB Size S3 sink connector MSK Connector task creat dag create more efficient build faster build smaller build image Multi-stage build S3 Connector kafka authentication sasl/scram Service Account Cloudwatch Filter Pattern Slack Alarm Private Node Group Instance Type 변경 Spot Instance 예약인스턴스 Kafka Console Consumer Kafka Console Producer AWS MSK Kafka 기능 Kafka usecase kafka 장점 Kafka 구조 Grep Filter 재귀 쿼리 Recursive query KeyedStream DataStream Kinesis Data Analytics S3 tier S3 Storage Class Airflow 구조 savings plan delta lake Mapping Update Index Mapping Dynamic Inventory Kinesis Datastreams TSDB RDS Proxy Presigned URL ServiceAccount Lambda Module Lambda Layers sidecar container Mangum AWS SAM Kafka Connector peering connection fluent-bit Reserved Instance remote state maxBy Container Image EC2 instance fast api with recursive Aggregations cloud function kubeconfig tfstate S3 Glacier 탄력적 IP Mac M1 Firehose kubernets first_value boto3 ConfigMap 데이터 엔지니어링 helm chart fastapi helm IOT Core AWS CLI Elastic IP NAT Gateway serverless docker-compose rolebinding k8s RBAC API Gateway Inventory flatMap Datalake parquet dynamodb Asyncio Apache Flink conda reindex Google Cloud Platform 데이터 아키텍처 SQLAlchemy ansible Redshift package management Elasticsearch 윈도우 함수 window function Reduce REST API bitbucket table size coroutine Athena exclude xcom prometheus PANDAS timetable 데이터 수집 row_number DAG 트랜잭션 RI efs iam metric flush fetch 가상환경 SUM Pull gcs Stash permission ECS time travel EIP Regex parser DB Connection RESTORE ETL 책 리뷰 Role AUTOCOMMIT UTC rollback grep monitoring Push PostgreSQL copy commit poetry acid Filter amazon Lake House map secret Dashboard API history Linux ubuntu include