Docker compose 사용해서 Flink 설치

docker compose 사용해서 Flink 설치

• docker-compose.yml

version: "2.2"
services:
  jobmanager:
    image: flink:latest
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager        

  taskmanager:
    image: flink:latest
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2        
  sql-client:
    image: flink:latest
    command: bin/sql-client.sh
    depends_on:
      - jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        rest.address: jobmanager

• Flink 클러스터 배포/시작

docker-compose up

• Web UI 접속
  •  http://localhost:8081 로 접속 가능
• job-manager 접속

docker exec -it $(docker ps --filter name=jobmanager --format={{.ID}}) /bin/sh

• 작업 제출 - 예제 파일 실행

./bin/flink run ./examples/streaming/WordCount.jar

-> docker 로그에 결과값 출력되는것 확인 가능

...
jobmanager-1   | 2025-01-13 15:22:04,016 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - counter -> Sink: print-sink (1/1) (a1f1bc787405291b8a0e261d8a26f08b_90bea66de1c231edf33913ecd54406c1_0_0) switched from INITIALIZING to RUNNING.
jobmanager-1   | 2025-01-13 15:22:04,041 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: in-memory-input received split request from parallel task 0 (#0)
taskmanager-1  | (to,1)
taskmanager-1  | (be,1)
taskmanager-1  | (or,1)
taskmanager-1  | (not,1)
taskmanager-1  | (to,2)
taskmanager-1  | (be,2)
taskmanager-1  | (that,1)
...
taskmanager-1  | (more,1)
taskmanager-1  | (and,3)
jobmanager-1   | 2025-01-13 15:22:04,080 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Clearing resource requirements of job 4b72ce76bd4144918a2dc5d25191b119
taskmanager-1  | (by,2)
jobmanager-1   | 2025-01-13 15:22:04,081 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job WordCount (4b72ce76bd4144918a2dc5d25191b119) switched from state RUNNING to FINISHED.
taskmanager-1  | (a,2)
jobmanager-1   | 2025-01-13 15:22:04,081 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Stopping checkpoint coordinator for job 4b72ce76bd4144918a2dc5d25191b119.

• 실행중인 작업 확인

./bin/flink list

• 특정 작업 취소

./bin/flink cancel <job-id>