Deploying Flink on Kubernetes - David Anderson

© 2019 Ververica
David Anderson | @alpinegizmo | Training Coordinator
Getting Started with
Apache Flink® on Kubernetes

2 © 2019 Ververica
About Ververica
Creators of
Apache Flink®
Real Time
Stream Processing
for the Enterprise

3 © 2019 Ververica
Outline
1. Introduction
2. Detailed Example
3. Debugging Tips
4. Future Plans

4 © 2019 Ververica
Why Containers?
• Containers provide isolation at low cost
– Require fewer resources than VMs
– Smaller, boot faster
• Simpler to manage
– Each container does one thing
– Consistent packaging
• Enables flexible and dynamic resource allocation
– Scalable
– Composable

5 © 2019 Ververica
Container Orchestration with Kubernetes
• Declarative configuration:
– You tell K8s the desired state, and a background process makes it happen
• 3 replicas of this container should be kept running
• A load balancer should exist, listening on port 443, backed by container with this label
• Core resource types:
– Pod: a group of one or more containers
– Job: keeps pod(s) running until finished
– Deployment: keeps n pods running indefinitely
– Service: a REST object backed by a set of pods
– Persistent Volume Claim: storage whose lifetime is not coupled to any of the pods

6 © 2019 Ververica
Vision: Flink as a Library
• Makes deployments simpler
– Focus is on deploying/running an application
– You build one, complete job-specific Docker image that includes:
• Your application code
• Flink libraries
• Other dependencies
• Configuration files

7 © 2019 Ververica
Flink’s Runtime Building Blocks
• Cluster framework-specific
• Manages available TaskManagers
• Acquires / releases resources
ResourceManager
TaskManagerJobManager
• Registers with ResourceManager
• Provides “task slots”
• Assigned tasks by one or more JobManagers
• One per job
• Schedules job in terms of "task slots"
• Monitors task execution
• Coordinates checkpointing
Dispatcher
• Touch-point for job submissions
• Spawns JobManagers

8 © 2019 Ververica
Flink’s Runtime Building Blocks
• Cluster framework-specific
• Manages available TaskManagers
• Acquires / releases resources
ResourceManager
TaskManagerJobManager
• Registers with ResourceManager
• Provides “task slots”
• Assigned tasks by JobManager(s)
• One per job
• Schedules job in terms of "task slots"
• Monitors task execution
• Coordinates checkpointing
Dispatcher
• Touch-point for job submissions
• Spawns JobManagers

9 © 2019 Ververica
Runtime Building Blocks (on Yarn)
ResourceManager
(3) Request slots
TaskManager
JobManager
(4) Start TaskManager
(5) Register
(7) Deploy Tasks
Dispatcher
App/Client
(1) Submit Job
(2) Start JobManager
(6) Offer slots

10 © 2019 Ververica
But we’re not quite there yet with K8s

Flink on K8s: current status
• Still using the legacy standalone resource manager
• Deployment establishes a static execution environment
• You will have a k8s manifest that effectively says
– there should be n taskmanagers that look like this
Flink is not aware of Kubernetes

Master Container
ResourceManager
JobManager
Mini Dispatcher
(2) Run & Start
Worker Container
TaskManager
Worker Container
TaskManager
Worker Container
TaskManager
(3) Register
(4) Deploy Tasks
(0) One image is built that can be either a Master or Worker
(1) Container framework starts Master & Worker Containers
Flink job cluster on K8s

2. EXAMPLE
https://github.com/alpinegizmo/flink-containers-example

Very Simple Streaming Job
https://github.com/alpinegizmo/flink-containers-example
data generator RichFlatMap print
# events per user
keyBy

Desired Runtime Landscape for K8s

Steps
1. Build the docker image
2. Set up job cluster (k8s job) &
task managers (k8s deployment)
3. Set up job cluster service
4. Add minio for checkpoints

1: Build a docker image
ADD $flink_dist $FLINK_INSTALL_PATH
ADD $job_jar $FLINK_INSTALL_PATH/job.jar
. . .
COPY docker/flink/flink-conf.yaml $FLINK_HOME/conf
COPY docker/flink/log4j-console.properties $FLINK_HOME/conf
COPY docker/flink/docker-entrypoint.sh /
. . .
ENTRYPOINT ["/docker-entrypoint.sh"]
Dockerfile

. . .
JOB_CLUSTER="job-cluster"
TASK_MANAGER="task-manager"
CMD="$1"
shift;
if [ "${CMD}" == "${JOB_CLUSTER}" -o "${CMD}" == "${TASK_MANAGER}" ]; then
if [ "${CMD}" == "${TASK_MANAGER}" ]; then
exec $FLINK_HOME/bin/taskmanager.sh start-foreground "$@"
else
exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
fi
fi
exec "$@"
docker-entrypoint.sh

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: flink-task-manager
spec:
replicas: ${FLINK_NUM_OF_TASKMANAGERS}
template:
metadata:
labels:
app: flink
component: task-manager
spec:
containers:
- name: flink-task-manager
image: ${FLINK_IMAGE_NAME}
imagePullPolicy: Never
args: ["task-manager",
"-Djobmanager.rpc.address=flink-job-cluster"]
task-manager-deployment.yaml.template
apiVersion: batch/v1
kind: Job
metadata:
name: flink-job-cluster
spec:
template:
metadata:
labels:
app: flink
component: job-cluster
spec:
restartPolicy: OnFailure
containers:
- name: flink-job-cluster
args: ["job-cluster",
"-Djobmanager.rpc.address=flink-job-cluster",
"-Dblob.server.port=6124",
"-Dqueryable-state.server.ports=6125"]
ports:
- containerPort: 6123
name: rpc
name: blob
name: query
name: ui
job-cluster-job.yaml.template
2: K8s manifests

kind: Job
metadata:
spec:
template:
metadata:
labels:
app: flink
spec:
containers:
ports:
name: rpc
name: blob
name: query
name: ui
2: K8s manifests
kind: Deployment
metadata:
spec:
template:
metadata:
labels:
app: flink
spec:
containers:

kind: Deployment
metadata:
spec:
template:
metadata:
labels:
app: flink
spec:
containers:
kind: Job
metadata:
spec:
template:
metadata:
labels:
app: flink
spec:
containers:
ports:
name: rpc
name: blob
name: query
name: ui
2: K8s manifests

apiVersion: v1
kind: Service
metadata:
labels:
app: flink
spec:
ports:
- name: rpc
port: 6123
- name: blob
port: 6124
- name: query
port: 6125
nodePort: 30025
- name: ui
port: 8081
nodePort: 30081
type: NodePort
selector:
app: flink
3: Expose job cluster as a service
job-cluster-service.yaml
internal ports
external ports

4: Setup minio for checkpoints & savepoints
• S3-compatible storage service
• Apache License v2.0
• Lightweight, easy to setup

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minio-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
minio-standalone-pvc.yaml

minio-standalone-deployment.yaml
kind: Deployment
metadata:
name: minio
spec:
strategy:
type: Recreate
template:
metadata:
labels:
app: minio
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: minio-pv-claim
containers:
- name: minio
volumeMounts:
- name: data
mountPath: "/data"
image: minio/minio:RELEASE.2019-03-13T21-59-47Z
args:
- server
- /data
env:
- name: MINIO_ACCESS_KEY
value: "minio"
- name: MINIO_SECRET_KEY
value: "minio123"
ports:
livenessProbe:
httpGet:
path: /minio/health/live
port: 9000
initialDelaySeconds: 120
periodSeconds: 20

apiVersion: v1
kind: Service
metadata:
name: minio-service
spec:
type: NodePort
ports:
- port: 9000
nodePort: 30090
selector:
app: minio
s3.path-style-access: true
s3.endpoint: http://minio-service:9000
minio-standalone-service.yaml
flink-conf.yaml

/bin/sh -c "
sleep 10;
/usr/bin/mc config host add myminio http://minio-service:9000 minio minio123;
/usr/bin/mc mb myminio/state;
exit 0;
"
minio setup job
state.checkpoints.dir: s3://state/checkpoints
state.savepoints.dir: s3://state/savepoints
s3.access-key: minio
s3.secret-key: minio123
flink-conf.yaml

A Note on Bucket Addresses
• Two ways to specify buckets:
– virtual-hosted style: state.minio-service:9000
– path-style: minio-service:9000/state
• It’s easier to get path-style addresses working, by either using
– s3.path-style-access: true (requires flink 1.8+)
or by
– specifying the endpoint with its IP address, rather than hostname

Rescaling
$ kubectl scale deployment -l component=task-manager --replicas=2
deployment.extensions "flink-task-manager" scaled
$ flink modify 00000000000000000000000000000000 -p 8 -m localhost:30081
Modify job 00000000000000000000000000000000.
Rescaled job 00000000000000000000000000000000. Its new parallelism is 8.

3. DEBUGGING

. . .
JOB_CLUSTER="job-cluster"
TASK_MANAGER="task-manager"
if [ "${CMD}" == "${JOB_CLUSTER}" -o "${CMD}" == "${TASK_MANAGER}" ]; then
echo "Starting the ${CMD}"
echo "config file: " && grep '^[^n#]' $FLINK_HOME/conf/flink-conf.yaml
if [ "${CMD}" == "${TASK_MANAGER}" ]; then
exec $FLINK_HOME/bin/taskmanager.sh start-foreground "$@"
else
exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
fi
fi
exec "$@"
docker-entrypoint.sh

Starting the job-cluster
config file:
jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
taskmanager.heap.size: 1024m
taskmanager.numberOfTaskSlots: 4
parallelism.default: 1
high-availability: zookeeper
high-availability.jobmanager.port: 6123
high-availability.storageDir: s3://highavailability/storage
high-availability.zookeeper.quorum: zoo1:2181
state.backend: filesystem
state.checkpoints.dir: s3://state/checkpoints
state.savepoints.dir: s3://state/savepoints
rest.port: 8081
zookeeper.sasl.disable: true
s3.access-key: minio
s3.secret-key: minio123
s3.path-style-access: true
s3.endpoint: http://minio-service:9000
logs

4. FUTURE PLANS

Tighter Integration with K8s
• Active mode
– Flink is aware of the cluster manager that it is running on,
and interacts with it
– Examples exist, e.g., FLIP-6 YARN
• Reactive mode
– Flink is oblivious to its environment
– Flink may react to resources changes by scaling job

Active k8s Integration
K8s deployment
controller
Client
TaskManager
JobManager
K8sResourceManager
ApplicationMaster
TaskManager
(3) Submit job
(1) Submit AM deployment
(2) Start AM
pod
(4) Start JM
(5) Request slots
(6) Submit TM
deployment
(7) Start TM pod
(8) Register(9) Request slots
(10) Offer slots

FLINK-9953: Active Kubernetes integration
The ResourceManager can talk to Kubernetes to launch new pods

Reactive Container Mode
• Relies on external system to start/release
TaskManagers, e.g.,
– Kubernetes Horizontal Pod Autoscaler
– GCP Autoscaling
– AWS Auto Scaling Group
• Re-scale job as resources are
added/removed (take savepoint and resume
job with new parallelism automatically)
• By definition works with all cluster managers
Flink cluster
JM TM TM
ASG
Start new TM if
CPU% > threshold
Monitor metrics, e.g, CPU%
Register
& offer
slots
Event rate over time

FLINK-10407: Reactive container mode
Re-scale job as resources are added/removed

Summary
• Flink currently supports job and session clusters on K8s
• Example
– https://github.com/alpinegizmo/flink-containers-example
• Active K8s integration is in progress
• Reactive container mode has been designed/planned
• Call to action:
– Umbrella tickets: FLINK-9953, FLINK-10407
– Join discussions on dev@flink.apache.org

Thank you!

Questions?

www.ververica.com @VervericaDatadavid@ververica.com

Deploying Flink on Kubernetes - David Anderson

More Related Content

What's hot (20)

Similar to Deploying Flink on Kubernetes - David Anderson (20)

More from Ververica (20)

Recently uploaded (20)

Deploying Flink on Kubernetes - David Anderson