If you use a managed database service like AWS RDS or Google Cloud SQL, it is super easy to create a snapshot and create a new instance based on this snapshot, but there is an even easier way to clone a database. At Redactics our use case was buildin...
Dynamic task mapping (DTM) is a major feature that adds a lot of flexibility to how you build your DAGs. However, it does not provide infinite flexibility and break you free of being beholden to Airflow's patterns. For example, your task mappings are...
The most logical starting place is to stand up a new database cluster based on a snapshot, especially if you are using a managed database service like AWS RDS where doing so is easy. However, this still requires some manual steps or some sort of scri...
Ever wanted to actually know when an Airflow DAG Run has completed? Perhaps your use case involves this completed work being some sort of workflow dependency, or perhaps it is used in a CI/CD pipeline. I'm sure there are a myriad of possible scenario...
There are studies, including this one, that show that the vast majority of data breaches are due to human error. Some of this human error is the result of being vulnerable to manipulation, social engineering, etc. Other forms are due to bad practices...
So often throughout my career I've come across projects where seed data was created during the very early days of a repo, and the bare minimum is done to update this seed data as schema changes. If a new string column is added, maybe at best develope...
Virtually every backend engineer I've ever met has some sort of opinion about microservices. If I had to summarize the entire discussion: Not having a giant monolith that requires tactics for establishing code boundaries and preventing code paths fr...
We bet our business around our customers being sold on running our software in their own infrastructure, rather than building a more traditional SaaS platform. Redactics is all things data management: primarily data pipelining with PII removal for te...
Airflow's PostgresOperator includes a field called parameters for providing SQL parameters for prepared statements. We wanted to use named parameters since our parameter order would vary, which meant understanding the syntax. In their provided exampl...
