There are lots of great blog posts that dive into detail on how to work with variables in Apache Airflow. Instead, use the Apache Airflow cli either via the MWAA API or via the bash and python operators within a DAG You cannot use the Apache Airflow REST API within MWAA. You can check out the summarised list of Airflow commands that work here and you will notice that this does vary between Airflow 1.x and 2.x If you have been used to using the Apache Airflow cli to work with variables, then some of the Airflow commands may or may not be available to you. There are some thing to be aware of when working with variables and parameters in MWAA. finally, some additional ideas with example code, that provide some alternative approaches you can experiment with.third, how you can alter how your workflows behave by using parameters during runtime.second, how you can work with variables and parameters either via the Apache Airflow UI, the Airflow clip or in your DAGs.first, a look at the different options you have when working with variables and parameters in your workflows.In this particular post I want to look at how you can use variables in Apache Airflow, and more specifically MWAA to do a few things: Access to an AWS region where Managed Workflows for Apache Airflow is supported, as well as an environment of Amazon Managed Workflows for Apache Airflow already setup - you can follow my previous link here or check out the instructions in the GitHub repo which will walk you setting up an environment.An environment with the AWS CLI tools configured and running.An AWS account with the right level of privileges.I have a very short survey which you can complete here, and the first 20 will receive $25 AWS credit codes. Please let me know if you find this post useful. This is by no means an exhaustive list, and welcome feedback on other ways that I might have missed or are improvements on what is listed below.Īll the source code in this blog post can be found in this GitHub repository To address these, I thought I would put together this post to dive a little deeper into some of your options when looking to create parameterised and re-usable workflows (DAGs) that you can run on MWAA. Looking at some of the threads in the Apache Airflow slack channel, as well as some of the internal channels we use, it became clear that some guidance on how to do this when using Managed Workflows for Apache Airflow (MWAA) would help those, either new to MWAA or coming to MWAA from self managed Apache Airflow.Īs I looked into this area, I noticed that there was also more interest in the integration of MWAA with AWS Secrets Manager as a way of storing and retrieving those variables securely. This makes a lot of sense, as you may find that you repeat similar tasks in your workflows, and so this approach allows you to maximise the re-use of that work. Maximising the re-use of your DAGs in MWAAĭuring some recently conversations with customers, one of the topics that they were interested in was how to create re-usable, parameterised Apache Airflow workflows (DAGs) that could be executed dynamically through the use variables and/or parameters (either submitted via the UI or the command line).

Aws apache airflow