Story Tag: airflow

Stories Tag: airflow

Data Engineering at Dwelo. A reflection of my three months… | by Anusha Karunakaran | Sep, 2020 | Medium

admin 2 weeks ago in

A reflection of my three months Journey. Data is stored in a distributed manner. It is highly important to build a reliable data pipeline to deal with batch, incremental loads of data. Dwelo uses bleeding edge of data warehouse technology in architecting distributed systems , creating reliable pipelines and collaborating with data science teams to build the right solutions for them. Here’s a brief description…

The journey of deploying Apache Airflow at Grab | by Grab | Sep, 2020 | Medium

admin 2 weeks ago in

At Grab, we use Apache Airflow to schedule and orchestrate the ingestion and transformation of data, train machine learning models, and the copy data between clouds. There are many engineering teams at Grab that use Airflow, each of which originally had their own Airflow instance. The proliferation of independently managed Airflow instances resulted in inefficient use of resources, where each team ended up solving the…

ML code delivery to production in a few days: how we saved a lot of our company money with a...

admin 3 weeks ago in

In our company in the production we have a hundreds of scheduling machine learning (ML) models and calculation scripts on a constant basis. And our Data Engineering division had taken a task to make some CD/CD component that was most suitable for our goals: For achieving our goals we selected Kubernetes as best Docker orchestrating system, Airflow as one of the fast upcoming schedulers and…

Video: Big Data on Kubernetes – Make It Flow (Hebrew) | by Riskified Technology | Riskified Technology | Aug, 2020...

admin 4 weeks ago in

Apache Airflow is a leading framework for managing data workflows. It’s pretty good out of the box, but we think running it on K8s makes it even better! It takes out the tedious work and lets you focus on orchestrating and scheduling pipelines.In this talk, Lior shares our story of how we chose and deployed Airflow on k8s, the challenges we faced along the way,…

Airflow on Kubernetes with Git Sync | by Shuyu Wang | Aug, 2020 | Medium

admin 1 month ago in

This article will show you how we deployed airflow on Kubernetes while using git sync and kubernetes executor. Our airflow deployment includes the following components: This configmap includes the airflow.cfg which helps us set up kubernetesExecutor and remote s3 log. To create Volume and volumeMounts for the worker pod, we created a template file and put it in the image and specify the pod_template_file. The…

Introduction Cloud, Apache Airflow and BigQuery for Data Engineer from Beginner | by Top’p Kullawattana | Jul, 2020 | Medium

admin 2 months ago in

ในบทความนี้ ผมจะพาทุกคนมาทำความเข้าใจกับการใช้ Cloud ในการทำงานด้าน Data Engineer ซึ่ง Cloud มีส่วนช่วยในการจัดการ Data และในปัจจุบัน Cloud Platform มีบริการหลายรูปแบบ เช่น VM, Kubernetes, Serverless, Database, Data Warehouse, CI, Load Balance โดยมี Data Center ในหลายประเทศ โดยชนิดของ Cloud ประกอบด้วย Public Cloud หรือ Major Cloud Platform Providers เช่น Google Cloud, AWS, Azure, IBM Cloud, Alibaba Cloud, Tencent Cloud และยังมี Private Cloud กับ Hybrid Cloud ด้วยนะ คุณสมบัติของ Cloud แบบต่างๆ การสร้าง Hybrid Cloud มีคุณสมบัติ ดังนี้ การใช้…

Machine Learning in production (2) — large scale ML training

admin 3 months ago in

In a previous post (here), I shared some experience on how to introduce a structured Python project for machine learning as early as the first working notebook. By leveraging a powerful platform like Databricks, you can have pretty fast feedback loops. This approach works relatively well for a small scale machine learning project, but you will probably need something more for a large scale machine…

Run Airflow and Jenkins as a Kubernetes way to saving cost

admin 3 months ago in

今天要一次來談兩個服務，分別是 Jenkins 跟 Airflow，大家印象中可能覺得，這兩個服務常被使用的場景，有些許的不一樣，Jenkins 是歷史悠久的老頭，大部分人用它來完成一些 CI/CD 的工作，而 Airflow 則是近年來大家在資料工程領域常用的 ETL 工具。 漸強實驗室使用這兩個服務已經有蠻長一段時間，不意外的 Jenkins 用來負責各種服務的 CI/CD pipeline，每週在環境中自動化的迭代與發佈數十次的版本，而 Airflow 則是負責定時在各個資料庫與資料倉儲中提取、搬運以及計算資料。 今天要來分享（推坑）的，並不是叫大家使用這兩個好棒棒的服務（因為我相信這兩個服務，在各自的領域都是非常多人使用的解決方案），而是推薦大家如何使用🤔！？ 會根據我在漸強實戰的服務架構與系統維運經驗，談談漸強在系統架構層面，是用如何用 Kubernetes 的姿勢（方式）來部署和使用 Jenkins & Airflow，也會談到這樣的選擇帶來了哪些好處？跟為什麼如果你的團隊也熟悉 Kubernetes ，非常值得推薦你們也嘗試看看。 補充說明一下，Jenkins & Airflow 的配置與運作模式是蠻不一樣的，所以此文章並不分享配置細節（不然寫不完…），而是要來用比較大局觀的方式聊聊一些架構跟維運上的概念，以及我的一些想法。 如果你熟悉這兩個服務（風流＋老爺），說到底！這兩個服務是有一些本質上的相同，那就是： 就是他們都是根據某些規則，來觸發調度一系列的任務 以規則來說， Jenkins 可能是你的 PR Merge 後，從 Github 打來的 webhook，而 Airflow 就是 Scheduler 的時間排程；再來是則是觸發之後要調度的任務，任務簡單的說就是跑一個或是一組程式，Jenkins 可能就是跑你所撰寫的 pipeline，而 Airflow 則是執行你定義的 Dag。 所以每當條件觸發了某項任務，相對就會需要有一個環境與跟運算資源來完成這個任務。要環境！？要資源！？不就正剛好是 Kubernetes 的拿手好戲嗎？ 一直以來 Kubernetes 被定義成 Container Orchestration，可以用來快速方便的調度部署許多容器化的服務，換個角度看，利用容器化技術可以達成輕量快速的環境切分，加上動態的容器生命週期控制，換句話說 Kubernetes 就是一個環境與資源的調度管理器，可以動態且快速地提供不同的環境與運算資源。 你可能會說，我操作一些 Cloud 的 API…

Deploying Scalable, Production-Ready Airflow in 10 Easy Steps Using Kubernetes

admin 3 months ago in

Airflow, Airbnb’s brainchild, is an open-source data orchestration tool that allows you to programmatically schedule jobs in order to extract, transform, or load (ETL) data. Since Airflow’s workflows are written in Python as DAGs (directed acyclic graphs) they allow for complex computation, scalability, and maintainability unlike cron jobs or other scheduling tools. As a data scientist and engineer, data is incredibly important to me. I…