Read the invoice

My lakehouse runs on Yandex Cloud and has for months. A serverless container wakes up every morning, pulls the source workbooks from one external endpoint, stages them to Parquet, and publishes eight bronze → silver → gold models. A payroll process depends on the last one. Nobody asked me to move it.

I rebuilt it on AWS anyway, as a twin: a second implementation of the same pipeline, reading the same source files, in a separate account, with production left strictly untouched. The Terraform lives in its own directory with its own state, and the CD workflow that deploys production still points at the old one and needed no change.

The goal was never migration. It was to learn AWS on something with consequences. A greenfield tutorial teaches you the happy path; a twin gives you an oracle. When the AWS numbers disagree with the Yandex numbers, one of them is wrong, and finding out which is the entire education.

I was also honest with myself about the trade, because it came up almost immediately when choosing a compute engine. The question in that decision was not which engine is better. It was whether I was buying a pipeline or an education — and it is fine to say education, as long as you say it out loud.

The third commit in the twin’s history created a monthly budget and a cost anomaly monitor. The first table came four commits later.

That ordering was deliberate, and it is the first thing I would tell anyone starting on AWS. The platform has no spending ceiling. There is no setting that stops the bill, only settings that tell you about it, and they are worth having in place before the first resource that can run away from you.

Two other guards went in at the same time, both cheap and both load-bearing:

• An account guard. My shell’s [default] profile pointed at a different AWS account. The provider sets allowed_account_ids, so a run with the wrong credentials aborts before creating anything, rather than building the sandbox in the wrong place. This is four lines and it has already saved me once.
• Default cost-allocation tags. Every resource is tagged Project, ManagedBy and Repo automatically by the provider. The comparison this whole project exists to produce therefore became a Cost Explorer query instead of an archaeology project.

The budget ceiling was $20 a month. That number did real work later — it is what disqualified managed Airflow in about ten seconds of arithmetic, and it is why the twin has no VPC.

AWS will run something on a schedule in at least seven ways. Choosing between them was the single most valuable hour I spent on this project, and the value was almost entirely in working out what to judge them on. They barely differ on firing a job. They differ enormously on whether they ever tell you what happened.

That question splits into three failures which are genuinely distinct:

1. The job ran and failed. Easy, and identical everywhere — it is job-level telemetry, indifferent to whatever started the run.
2. The start was rejected. Harder. StartJobRun can throw ConcurrentRunsExceededException, and most options drop that on the floor.
3. It never fired at all. The hard one, and nothing on AWS reports it. A trigger left in CREATED, a schedule deleted by a bad apply, an end date in the past — all produce perfect silence.

Every option is loud about the failure you would have noticed anyway, and quiet about the one you would not.

Six of the seven cost nothing. The decision was therefore never about money — it was about which failures each one would let pass in silence, and what it would take to close that gap myself.

I shipped a plain Glue trigger, which is the option with the worst telemetry about itself: no CloudWatch metric, no EventBridge event, no last-fired timestamp. The tell is that AWS built a “Scheduled Crawler Invocation Failure” event for crawlers and never shipped a job equivalent. I chose it anyway, because the scheduler belongs next to the thing it schedules — no second service, no extra IAM role, no derived ARN string — and because the monitoring gap is fixable.

Fixing it is the part I actually learned something from. There is no metric for “did not run”, so you cannot alarm on one. You have to manufacture absence: build a rule that matches only SUCCEEDED, point it at a log group so it has a target, and then alarm on that rule’s TriggeredRules metric going missing.

The detail that decides whether any of this works is how CloudWatch reads a gap. By default it treats a missing datapoint as unknown rather than bad, so the alarm goes to an inconclusive state instead of firing. You have to tell it explicitly to read emptiness as failure. Leave the default in place and the whole construction is decoration — a monitor that stays silent in exactly the situation it was built for.

One practical warning: a brand-new dead-man’s switch always fires once. Its lookback window is empty the moment you create it, and empty means failed. I predicted this, then measured it. It is worth treating as a free end-to-end test of the notification path — rule to metric to alarm to inbox — and expecting it again every time you rebuild the alarm.

A metric that only exists when things go right is the only honest way to detect nothing happening at all.

I ran four statements against a real table — create, insert, update, alter — and then downloaded and read every byte they left behind. The data files turned out to be the boring part.

Iceberg is not a service and not a file format. It is a specification for metadata files that sit beside ordinary Parquet and turn a set of objects into a table with transactions. Those metadata files form a chain five levels deep, which looks like over-engineering until you notice that each level exists to avoid rewriting, and avoid reading, the level below it.

Four things from that exercise changed how I think about table formats.

The catalog pointer is the entire transaction. S3 has no atomic rename. A single PutObject is atomic, but “make version 4 current instead of version 3” is not. Iceberg delegates its whole transactional guarantee to one compare-and-swap on that single parameter. Committing a write means swapping one value — which is also why pointing an engine at a path is not a catalog: a filesystem catalog infers the current version by listing, which is neither atomic nor safely ordered under concurrent writers.

Read the plurals. It is schemas, not schema; partition-specs, not spec. Iceberg keeps every schema and every partition layout the table has ever had. That is what buys time travel returning the schema of that era rather than today’s, and it is what makes partition evolution work — old files keep their old spec, new files use the new one, and a query spanning both handles each correctly.

Every bound is keyed by field id, never by column name. Rename a column and nothing in the manifests changes. This is the one-line explanation of why Iceberg renames are safe and Hive renames silently corrupt tables.

The metadata costs more than the data, at this scale. 6.7 KiB of Avro to describe a 770-byte Parquet file. That inverts as tables grow, and the inversion is the whole point: it buys three chances to skip before reading a single byte of data — partition ranges in the manifest list discard a whole manifest unopened, column bounds in the manifest discard a data file unopened, and the Parquet footer’s row-group statistics discard part of a file already selected.

The nicest surprise was the UPDATE. It left three files where there had been one, and the unexpected one has a schema of exactly two columns, file_path and pos. It is a position-delete file — the update did not update anything, it added a tombstone. Its field ids sit at the very top of the signed 32-bit range, so they can never collide with a user column.

The twin has no VPC. That decision saved more money than every optimisation I made put together.

A NAT gateway in Frankfurt costs $0.052 an hour, which is about $38 a month sitting completely idle, plus $0.052 per gigabyte processed on top of normal data transfer — so traffic through it is effectively billed twice. It also lives in a single availability zone, so a genuinely highly-available setup runs one per zone: roughly $114 a month before a single packet moves. Against a $20 ceiling, one NAT gateway would have consumed the entire platform budget nearly twice over.

And there are at least four cheaper rungs that most tutorials skip straight past:

What made the decision easy was asking what a VPC is actually for. A private subnet exists so that nothing on the internet can reach what is inside it. The twin has no database in a private subnet, no VPN, nothing on-premises, and nothing listening. It reaches exactly one thing outside AWS — a single endpoint that hands over the source files, over HTTPS, once a run. Everything else it touches is an AWS service on a public endpoint with IAM doing the authorisation.

The requirement a VPC exists to satisfy simply was not present. So the line item never appeared.

The mistake is not choosing NAT. It is choosing NAT without noticing that the requirement it solves was never in the design.

Before comparing engines I had to untangle a naming problem that costs people real confusion: three different products are called Glue. The Data Catalog is a metastore — an API holding table definitions, Hive-compatible and deliberately engine-neutral, which Athena, Trino, Flink, Snowflake and DuckDB all read. Glue ETL is managed Spark, a compute service with separate pricing. Crawlers and Studio are tooling around those. Choosing the catalog does not choose the engine; the two decisions are independent.

On the engine itself, the honest answer was not Spark, and it was not close.

This pipeline publishes 644 thousand rows, about two megabytes a table. A distributed engine has nothing to distribute. Spark’s entire value proposition — partition the data, parallelise across workers, shuffle over the network — describes work that does not exist here. The join that matters fits in cache. DuckDB would have kept 263 lines of working SQL, 8 data contracts, 31 checks, 10 foreign keys and a 97-test suite that runs offline in 27 seconds, and it would have cost a third as much.

There were specific traps on both sides, and they were more interesting than the cost difference:

• Spark writes too many files. One output file per partition per task. On two megabytes with default parallelism that is a scatter of tiny files — precisely what Iceberg then needs compaction to undo. Fixable with a coalesce, if you know to do it.
• Spark cannot parse the workbooks. The sales layout is classified by openpyxl outline levels — workbook structure, not cell values. Ingestion stays a general Python process in every possible design, so a Spark answer is a hybrid whether you planned one or not.
• DuckDB cannot create Iceberg tables. Inserts into existing tables commit fine through the Glue endpoint; CREATE TABLE returns a 400. Table creation has to come from Terraform or Athena — which is arguably the right seam anyway.
• DuckDB sizes itself wrong. Left unconfigured it reads the host’s memory and cores. Measured at 49.7 GiB and 12 threads on my laptop, against a single-core container. In a constrained runtime that must be pinned explicitly.
• Spark tests need Spark. Reproducing a 27-second offline suite for PySpark means a local Spark or the Glue Docker image, both an order of magnitude slower.

I shipped Spark. Reading about partitioning is not the same as having debugged a skewed join, Spark is on every data-platform job description, and the stated goal of the project was the education. That is a legitimate reason, and the cost of it is visible in the numbers: of a 404-second run, about 70 seconds is Spark starting up before any of my code executes. I measured 87 billed seconds to load 22 rows.

That overhead is also why the twin runs one Glue job for all eight models rather than one job per model. Eight jobs would pay the startup cost eight times, and sharing a single session preserves the all-or-nothing publish that the original has and per-table Iceberg commits would quietly give away.

My first cost comparison said AWS would be 17× more expensive. The measured answer is 2.6×. The error was not in the arithmetic. It was in the method, and this is the thing I actually learned.

I built that first estimate the way everyone builds one: inventory the resources on the Yandex side — buckets, images, container memory, runtime — multiply by published list prices, and total it up. That reconstruction accounted for 38 ₽ of a 119 ₽ bill. It was not wrong about any resource it looked at. It was wrong because it only looked at resources.

Then I read the actual itemised invoice.

Compute is free. Storage is nearly free. The pipeline’s actual work — 273 seconds a day of Spark-free pandas — costs nothing at all. What costs money is asking the object store where the files are.

That is 140,750 LIST requests a month, or 4,624 a day, against a free allowance of 10,000 a month. The arithmetic reconciles exactly: (140.75 − 10) × 0.5692 = 74.42 ₽.

They come from the shape of the warehouse. It is loose Parquet in prefixes, and DuckDB reads it by globbing — read_parquet('s3://…/dt=*/*.parquet'). Every glob is a LIST against the bucket, every model does several, and the pipeline runs daily across a lake of 4,710 objects.

Nothing about this is visible from the code. It reads as one line of SQL. The cost lives entirely in how the engine resolves a wildcard, and it is the single largest thing on the bill.

A cost model built from the things you own will miss the things you do.

The reversal

An earlier draft of my notes ended on an irony. Iceberg makes the AWS lake 32× smaller — 0.26 GB across 708 objects against 8.33 GB across 4,710 — and storage was the one thing Yandex barely paid for. Efficiency landing exactly where the money was not.

That reads completely differently now. An Iceberg table is never listed. The manifest names every file, so the engine opens a metadata pointer and goes straight to the data. There is no wildcard to resolve and no prefix to scan.

Which is why the entire AWS S3 line — storage and every request together — is $0.011 a month, against 87 ₽ of Object Storage on the Yandex side. Iceberg’s metadata layer turns out to be a direct answer to the largest line on that bill.

It still does not make AWS cheaper overall, because Glue has no free tier and charges $3.01 for work Yandex does for nothing. But the trade is now legible: AWS pays for compute to avoid paying for coordination. Yandex does the reverse.

Three things nobody was watching

Reading the invoice also turned up about 25 ₽ — a fifth of the bill — that is not the cost of running the pipeline but the cost of never deleting anything.

• The registry. 32 container images, 6.22 GB, the oldest more than three months old. CI pushes one per merge to main and nothing prunes them. Container Registry has no free tier; it bills from the first byte and grows with every deploy. Pure waste, and the easiest line to delete.
• The secret. 19.26 ₽ for 703 secret-version-hours — the second largest line on the bill, for a secret simply existing. Worth knowing that a stored secret has a standing monthly cost there, and AWS Secrets Manager charges a flat $0.40 for the equivalent — neither cloud gives you a free secret.
• The compute cliff. The container is provisioned at 4096 MB and peaks at 663 MB — a 6× gap — and billing is on allocated, not used. The 4 GB exists because a monthly full rebuild was OOM-killed at 2048 MB, so the daily run pays the monthly job’s peak every single day.

None of those three appear in any architecture diagram.

A port is only finished when you can prove it gives the same answers. Contracts, not-null checks and foreign keys can’t show that; they only prove a pipeline agrees with itself. So I rebuilt both sides on the same day from the same source files and compared the final table row by row.

19,404 rows on both sides, none missing from either. The key columns were identical on every row. The 174 rows that did differ all traced back to two changes I had made to the twin’s logic on purpose, not to anything that happened in the move between clouds.

That diff is the regression test I trust most.

Contracts cannot catch a wrong number that is well-typed and non-null. Only a second opinion can.

AWS runs the same pipeline for $3.44 a month against Yandex’s 119.25 ₽ — call it $1.30. About 2.6×, or roughly two dollars more, for verified-identical output.

Measured at 404 seconds a run, 2 × G.1X at $0.44 per DPU-hour, in eu-central-1. No free tier — the meter starts at the first second, and about 70 of those seconds are Spark starting up.

What the extra two dollars buys is not in dispute: Iceberg, Athena, a real catalog, time travel, schema evolution — and, as it turned out, no LIST bill.

Three things I will carry to the next AWS project:

1. Read the invoice. Every estimate I built by reasoning about resources was roughly right about the resources, and the bill was still triple the estimate, because the largest line came from a behaviour that no inventory contains and no diagram shows. The model will always be a model.
2. Ask what requirement a resource satisfies before you provision it. NAT gateway, a VPC, a lock table, S3 versioning on an Iceberg lake — each is the standard answer to a question this project never asked. The default is frequently right and frequently expensive.
3. Build the detector for the silence, not just the error. The failures that cost you are the ones that emit nothing. A metric that exists only on success, alarmed on its absence, is the cheapest honest monitoring there is.

The twin still has gaps I know about. Nothing expires Iceberg snapshots or compacts small files yet, so the lake grows without bound; the Terraform is applied from a laptop rather than CI; and Lake Formation is designed for but not built. Those are the next things, and I would rather write them down than pretend the thing is finished.