Moving 60 AWS jobs onto one Databricks platform

Starting point

A payments company of around 300 people. Three teams ran their own Glue jobs, two long-lived EMR clusters and a set of cron scripts, all reading the same S3 buckets. The same card transactions were transformed in four different ways, so revenue in the finance report never matched revenue in the product dashboard. Finance closed the month by reconciling the two by hand, which took six working days.

Decisions

We kept a short decision log during the project. These three entries shaped most of it.

1. Migrate in waves, not in one cut-over

The client asked for a single switch-over weekend. We pushed back: with 60+ jobs and no tests, a broken number would only show up at month-end. Instead, each wave of jobs ran on both platforms for two weeks, and an automated row-level diff had to come back clean before the old job was switched off.

Cost of this choice: about three extra weeks of double compute.

2. One owner per layer

1. 01S3 raw
2. 02Bronze
3. 03Silver
4. 04Gold
5. 05Finance & BI

Bronze and Silver went to the platform team, Gold to the analysts who use it. Before, all three teams wrote to every layer, which is how four versions of revenue had appeared in the first place.

3. Reconciliation logic in dbt, not in notebooks

Notebooks would have been faster to migrate. We rewrote the reconciliation rules once as dbt models with tests, because this was the logic finance needed to trust. It took longer than planned: two of the rules turned out to contradict each other, and finance had to decide which one was correct.

Before and after

	Before	After
Production jobs	63	18
Definitions of revenue	4	1
Month-end close	6 working days	2 working days
Monthly compute spend	100%	58%

Most of the compute saving came from shutting down the always-on EMR clusters, not from Databricks itself.