How We Loaded 65 Million Records into Azure Cosmos DB Over a Weekend — And Lived to Tell the Tale
Data migrations are rarely exciting — until something goes wrong at 2am on a Saturday. This is the story of how we pulled off a massive data migration over a single weekend, moving 65 million records into Azure Cosmos DB without losing a single row — and in doing so, cut our data costs by over 99%. Whether you're planning something similar or just curious about what a weekend war-room looks like, here's everything we learned.
The Problem: A Mountain of Data and a Ticking Clock
An external vendor handed us a dataset of 65 million records that needed to live in our Cosmos DB instance — fast. The catch? Our database was configured at just 1,000 Request Units per second (RU/s), which is essentially the "speed limit" for how fast Cosmos DB can process operations. At that rate, we ran the numbers and quickly realised we were looking at days of loading time. That wasn't going to work.
We needed a smarter approach.
Building the Pipeline: From Vendor to Database
Before any data could land in Cosmos DB, it had to travel through several stages. Here's how we set things up.
Figure 1 — End-to-end data pipeline architecture
The data arrived via SFTP. The vendor transferred the file securely to our Azure SFTP server — a clean, reliable handoff point that kept the data protected in transit.
From SFTP, it moved to Azure Data Lake. Using Azure Data Factory, we orchestrated the transfer into Azure Data Lake Storage. Think of this as our staging area — a highly scalable space where we could safely hold and process the data before loading it into its final destination.
We organised it into a Delta table in Databricks. This was a critical step. Delta tables give you transactional reliability, schema enforcement, and the ability to query and manipulate huge datasets efficiently. All 65 million records were neatly structured here, ready to be pushed into Cosmos DB in a controlled way.
Unlocking Performance: The RU/s Strategy
Cosmos DB measures throughput in Request Units per second (RU/s). Every read, write, and query consumes RUs, so your allocation directly dictates how fast data can flow. At our default of 1,000 RU/s, each write was fighting for a tiny slice of capacity.
We worked with our platform team to temporarily scale up to 10,000 RU/s — a 10x increase — purely for the duration of the migration. The results were dramatic. What would have taken several days was compressed into a 30-hour window.
Once the historical load was complete, we settled on 2,000 RU/s as our new steady-state — enough to handle real-time read traffic comfortably without over-provisioning. The performance has been excellent: fast query responses, no throttling, and a smooth experience for our application layer.
Figure 2 — RU/s journey: from baseline through migration to steady state
Key takeaway: Cosmos DB's throughput is elastic. You're not locked into your everyday configuration for one-off, high-demand tasks. Scale up for the burst, then find your optimal cruising altitude — it's exactly what the service is designed for.
Dynamic Scaling: Matching Throughput to Demand
Once the historical load was behind us, we didn't just set Cosmos DB to 2,000 RU/s and forget about it. Our application runs daily data jobs — and those jobs have very different throughput needs compared to a quiet overnight period. So we introduced a simple but effective pattern: scale up before the job, scale back down when it's done.
Before a daily run kicks off, we increase the RU/s to match the expected write and read volume. Once the job completes, we dial it back to the baseline. This means we're never paying for capacity we're not using, and we're never throttled when we actually need throughput.
This kind of dynamic scaling is one of Cosmos DB's most underappreciated strengths — and it's a big part of why our monthly costs have stayed as low as they have even with regular operational workloads on top of the initial load.
Figure 3 — Daily RU/s scaling pattern: scale up for jobs, back down at baseline
Why We Chose Batching Over a Single Big Load
Rather than trying to push all 65 million records in one go (spoiler: this is almost always a bad idea), we split the dataset into 15 equal batches. Here's why that decision paid off.
Figure 4 — Batch loading strategy: 65M records split into 15 manageable batches
- Recovery without panic. If batch 9 had failed, we could restart from batch 9 — not from scratch. That's the difference between a minor setback and a catastrophic re-run.
- Visibility into progress. With a monolithic load, you're watching a progress bar and hoping. With batches, you have clear checkpoints you can monitor at each stage.
- Predictable resource usage. Batching kept our CPU, memory, and RU consumption steady throughout the weekend rather than spiking unpredictably.
- Easier debugging. If something looked off in the data, it was far easier to pinpoint which batch was responsible.
The entire 15-batch process ran over roughly 30 hours — starting Friday night and completing Sunday morning.
Keeping Secrets Safe: Our Security Approach
One thing we were firm on from the start: no hardcoded credentials, anywhere.
Figure 5 — Security architecture: secrets in Key Vault, config in Delta table
All sensitive connection details — specifically the Cosmos DB Client ID and Client Secret — were stored in Azure Key Vault. Databricks was configured with scopes linked directly to Key Vault, so credentials were retrieved dynamically at runtime. Nobody reading the code or configs could extract a usable secret.
Non-sensitive environment details (like endpoints, database names, and container names) lived in a Databricks Delta table, making them easy to update across environments without touching any code.
This separation of concerns — secrets in Key Vault, config in Delta tables — is a pattern we'd recommend to any team working with sensitive credentials in a cloud data pipeline.
The Real Win: Significant Cost Savings
Here's the part that got everyone's attention well beyond the engineering team.
Before this migration, our application relied on an external vendor API to fetch data on demand. Every call to that API had a cost attached, and at the scale we were operating, those costs added up considerably day after day.
Once the data was in Cosmos DB and our application switched to querying it directly, the impact was immediate and substantial. The reduction in running costs was significant — the kind of number that makes stakeholders take notice. The migration paid for itself within the first few days of going live.
For any team still relying on expensive third-party API calls to serve data that doesn't change frequently, this is the single most compelling argument for bringing that data in-house. The engineering effort is a one-time cost — the savings are ongoing.
The Moment of Truth: Data Validation
After 30 hours of loading, the most nerve-wracking part was confirming that every single one of those 65 million records had made it across safely.
We built a simple reconciliation check: count the records in the source Delta table, count the records in the target Cosmos DB container, and compare. The result?
| Source records | Target records | Difference | Status |
|---|---|---|---|
| 65,000,000 | 65,000,000 | 0 | ✓ SUCCESS |
Zero data loss. Every record accounted for.
What We'd Do Again (and What We'd Do Differently)
✓ What worked brilliantly
- Batching gave us control and peace of mind
- Temporarily scaling RU/s was a game-changer
- Dynamic scaling for daily runs keeps costs low
- Replacing vendor API with Cosmos DB delivered significant cost savings
- Delta tables made data rock-solid to work with
- Azure Key Vault kept credentials secure throughout
↻ What we'd refine next time
- Automate RU/s scaling rather than coordinating manually
- Build a real-time dashboard to track batch progress
- Consider parallel batches with enough RU/s headroom
The Bottom Line
A migration of 65 million records sounds intimidating — and honestly, it is. But with the right architecture, a smart batching strategy, and a willingness to temporarily spend a bit more on throughput, it becomes a very manageable engineering problem. And in our case, it turned out to be one of the best financial decisions our team made all year.
Don't fight your infrastructure — configure it for the task at hand. Cosmos DB is built to scale. Use that elasticity when you need it, find your optimal steady state once you're done, and make sure you're not paying a third party for data you could own outright.
If you're facing a similar challenge, we hope this gives you a solid starting point. Drop a comment below — we'd love to hear how others have tackled this.