Just clone and run make start-airflow to get started! Add some CI jobs to deploy your code and you're done. Airflow Repository Template - A boilerplate repository for developing locally with Airflow, with linting & tests for valid DAGs and plugins.Start Building Better Data Pipelines With apache Airflow 2020-Oct - Naman Gupta covers the basics of Airflow and its concepts.Kimaru Thagana covers a practical case of doing an ETL process using Apache Airflow using a dummy ecommerce store's transactional, user and product data. ETL with Apache Airflow for Data Analysis on Transaction Data.Introduction to Airflow - A web tutorial series by maxcotec for beginners and intermediate users of Apache Airflow.Apache Airflow Monitoring Metrics - A two-part series by maxcotec on how you can utilize existing Airflow statsd metrics to monitor your airflow deployment on Grafana dashboard via Prometheus.Airflow-Component - Lightweight installer of federated Airflow-Airflow (RabbitMQ) reference architectrure on Compute node(s).Introducing KEDA for Airflow - How to use KEDA scaler system to enable autoscaling of celery workers based on data stored in the Airflow metadata database.Distribute & deploy Apache Airflow via Python PEX files - Example repo with steps to bundle, distribute, & deploy Apache Airflow as PEX files.A CeleryExecutor docker-compose deployment is available here. Bitnami Airflow Worker Docker image - A secure and up-to-date docker image for Airflow Worker maintained by Bitnami.Bitnami Airflow Scheduler Docker image - A secure and up-to-date docker image for Airflow Scheduler maintained by Bitnami.Bitnami Airflow Docker image - A secure and up-to-date docker image for Airflow maintained by Bitnami.Astronomer Platform - Apache Airflow as a Service on Kubernetes.Integrating Apache Airflow with Apache Ambari - Mykola Mykhalov walks through using Apache Ambari to configure and deploy an Airflow instance.Running Airflow on top of Apache Mesos - Blog describing how to configure Mesos to run all of the Airflow componenents.airflow-cookbook - Chef cookbook for deploying Airflow.airflow-k8s-executor-on-GKE - A detailed tutorial to get a scalable, low maintenance airflow kubernetes executor environment deployed on Google Kubernetes Engine with helm.airflow-on-kubernetes - A guide on all relevant resources, scripts and projects that relate to running Airflow on Kubernetes.mumoshu's repository has not been recently updated, but there are numerous forks that may be based on more recent releases. kube-airflow - This repository contains both an Airflow Docker image (that appears to have been based on Puckel's work) and Kubernetes service definition. Deploys after a few clicks with CloudFormation. aws-airflow-stack - An AWS based Airflow cluster deployment with CeleryExecutor.It can be docker pulled at datagovsg/airflow-pipeline. airflow-pipeline - Airflow Docker container that comes preconfigured for Spark and Hadoop.Kubernetes Custom Operator for Deploying Airflow - Kubernetes Custom controller (also called operator pattern) for deploying Airflow on Kubernetes.It is regularly updated and closely tracks the official Apache releases.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |