Getting Started with Flink Resource Management Mechanism | Medium medium.com
By Song Xintong (Wuzang) Edited by Wang Wenjie (volunteer of Flink community) This article is composed based on the Apache Flink series live broadcasts and shared by Song Xintong, a senior development engineer at Alibaba. It helps developers understand Flink resource management mechanism in depth from three aspects: basic concepts, current mechanisms and policies, and future development directions. The components introduced in this article are related to Flink resource management. A Flink cluster consists of one Flink master and multiple task managers. The Flink master and task managers are process-level components. Other components are in-process components. As shown in Figure 1, a Flink master has one resource manager and multiple job managers. Each job manager in the Flink master manages a specific job separately. The scheduler component of the job manager schedules and runs all tasks in the DAG of the job, and sends resource requests. The slot pool component of the job manager owns all resources allocated to the job. The only resource manager in the Flink master schedules resources in the entire Flink cluster and interconnects with external scheduling systems. External scheduling systems refer to resource management systems such as Kubernetes, Mesos, and Yarn.
Report Story