Things I don’t want
to debug twice.

One feature, several changes

A feature needs a database migration, an API change, an update to a background job and a change in another service. Four pull requests, possibly in different repositories. The user only cares about one feature.

Now imagine that merging any PR into main immediately deploys it to production. The migration goes first. The background job is still running the old code. The API change is waiting for review. Everyone starts negotiating deployment order in chat.

"Don't merge yet." "Five more minutes." "Wait, did anyone stop the job?" Apparently Slack is now a release orchestrator. Nobody told Slack.

Integration is not delivery

A merge means the change is accepted into the codebase. A release means a compatible set of changes is ready to run in production. Those are different decisions.

Deploying every merge can work well when changes are backward compatible and services can be released independently. Feature flags and expand-and-contract migrations help. But if components must change together, merging one PR does not make the whole feature ready.

Why manual steps make it worse

I once included a data transformation in a migration. The plan changed: someone would run the operation manually instead.

There was an important dependency. A background job using the same data had to be stopped first. I called that out. The operations still happened in the wrong order, and the resulting data needed repair.

The SQL was not the hard part. Coordinating the state of the system was. A warning in a message is not a release control.

What the pipeline should enforce

CI should require the agreed checks before merging: tests, code style, relevant integration checks and review by the responsible code owners. Those checks should run automatically, not depend on someone remembering to ask.

Deployment needs its own controls. Decide which versions belong to the release, check their compatibility and make the sequence explicit. If a migration requires a background job to stop, that dependency belongs in the release procedure. Access to production should be limited to the people and workflows that need it.

This does not have to be a complicated system. It does have to be repeatable, auditable and capable of saying "not yet" without a message from the tech lead.

Takeaway

A green PR is ready to merge. It is not necessarily a complete release. Slack is a useful messenger, but a terrible deployment orchestrator.

The green report

During changes to a package build and release process, a dependent component stopped working. The investigation uncovered a less comfortable problem: tests covered the happy path, while some error handling was excluded from the coverage calculation.

The code had branches for failures. Those branches were meant to do something. Nobody had verified that they did it correctly.

Meanwhile, CI was green. A lovely shade of green, in fact.

I want tests, not a percentage

I am not for or against 100% coverage. I am for tests that verify the behavior we actually wrote down and implemented.

If a function catches an exception, what should happen next? If an API returns an error, what does the caller receive? If a retry fails, do we stop, retry again or leave the data in a half-written state?

Why write those branches at all if we never check their behavior?

Coverage only tells us what the coverage tool counted as executed. It does not establish that the assertions were useful, that the expected outcomes were checked or that excluded code is safe. A blanket 100% requirement also ignores legitimate exceptions.

The deadline problem

There is a reason teams end up here. Delivery dates get tighter, tasks pile up, and tests are an easy place to save half an hour today. The happy path passes, the pull request goes through, and the missing cases become tomorrow's problem.

At the same time, management may still demand a perfect coverage number. That encourages people to optimize the number instead of testing the behavior.

This is not about blaming one developer. Speed, cost and quality have real trade-offs. If the schedule leaves no room to test failure cases, an impressive metric will not create that missing time.

Tests belong where the behavior lives

Unit tests are useful for library behavior. They do not replace integration tests in applications using that library, or end-to-end tests for complete user and data-processing flows.

For a shared package, a dependency update can trigger pull requests in consuming repositories. Their CI then runs the tests that exercise the package in a real application. A library passing its own tests is necessary, not sufficient.

Takeaway

Test the behavior the code promises, including errors and edge cases. A coverage percentage is a measurement, not a substitute for engineering judgment.

Just a quick proof of concept

Consulting taught me to treat a proof of concept as code that is likely to reach production. My joking "99%" is experience, not an industry statistic.

The story is familiar. A client wants to validate an idea. Two weeks, a small service, one integration, a few test events. We show a demo. It works.

"Can we start using it?"

Of course. It already works, right?

A few months later, there are more integrations and real users. Eventually someone asks whether the same service can process 100 million events a day. And here comes Godzilla, running straight toward the architecture diagram.

The number in the title is illustrative. Event size, traffic peaks, dependency limits and failure handling matter much more than a daily total.

Minimal does not mean disposable

A PoC should answer a question with as little code as possible. Nobody needs a fleet of microservices to prove that one API integration works.

But "we'll rewrite it later" is not a technical design. Business logic tangled with transport code, connections with no clear owner, missing error handling and a data model built only for demo inputs are all expensive things to carry into production.

I would rather keep the scope small and the boundaries clear. Make it obvious where data comes from, who owns resources, what happens on failure and which shortcuts were deliberate. You can postpone scaling machinery without postponing basic engineering.

CI/CD belongs in the PoC, too

This is the part that gets forgotten most easily. The minimum delivery process should be planned when the PoC starts, not invented after the first production incident.

I want a reproducible build, a few meaningful automated tests (including failure behavior), basic code checks and a working pipeline from the repository to a test environment. Configuration and secrets must stay outside the code. The first release should use a clear, repeatable procedure.

That does not mean building an enterprise deployment platform in week one. It means avoiding the classic release plan of "copy the files, change a few values on the server and hope nobody notices."

There should also be a distinction between accepting code and releasing it. Even a tiny PoC can grow into a feature with multiple dependent changes. A merge should not become an accidental production deployment simply because that was the fastest workflow to configure.

How I approach it

First, decide what the PoC must prove. Then decide which shortcuts are safe if the code is still running a year later.

Build only what the experiment needs, but give it a basic CI/CD path, tests for its expected behavior and enough structure to change it without starting over. Write down the shortcuts you chose and the conditions under which they will stop working.

The PoC may be temporary. The code can outlive teams, architects and managers. Godzilla will not care that the original estimate was only two weeks.

Takeaway

Build the smallest thing that proves the idea. Just assume someone will ship it. Because someone probably will.

Problem

After upgrading to Airflow 3.3.1, one of the DAG list API requests became unexpectedly slow. A request using dag_runs_limit=1 was taking roughly 3.7 seconds.

The interesting part was dag_runs_limit=1. Without it, the request was significantly cheaper. With it enabled, Airflow needed to retrieve the most recent DAG run for every DAG displayed in the UI.

Query pattern

The relevant database access was effectively:

SELECT ...
FROM dag_run
WHERE dag_id = ?
ORDER BY run_after DESC
LIMIT 1;

The database already had indexes covering the individual columns, but those indexes did not match the complete access pattern: filter by dag_id, then find the newest row by run_after.

Fix

A composite index matching the query was added:

CREATE INDEX idx_dag_run_dag_id_run_after
    ON dag_run (dag_id, run_after);

The important part is the column order: (dag_id, run_after). The database can locate rows for one DAG and immediately walk the index in run_after order.

Result

No frontend changes. No Airflow configuration changes. No caching workaround. The slowdown came from a database access pattern without an index matching it.

Why dag_runs_limit=1 exposed it

Asking for one DAG run sounds cheap, but the actual problem is repeated lookup: the newest run for every DAG in the result set.

WHERE dag_id = ?
ORDER BY run_after DESC
LIMIT 1

A query that is only slightly inefficient once becomes very visible when it is repeated across a page of DAGs inside a single API request.

Takeaway

When debugging slow endpoints, “the query has indexes” is not enough. The useful question is: does an index match the complete access pattern?

For queries shaped like WHERE a = ? ORDER BY b LIMIT 1, separate indexes on a and b may still be insufficient. A composite (a, b) index can make the difference.

In this case: ~3.7 s → < 0.7 s. From one index.

Problem

I needed to grant permission to reload dictionaries while limiting the privilege to only a couple of databases. The obvious attempt was to scope the grant:

GRANT SYSTEM RELOAD DICTIONARY
ON analytics.*
TO some_role;

That does not work, because this privilege is not defined at database or table level.

Check the privilege level

ClickHouse exposes privilege metadata through system.privileges. For SYSTEM RELOAD DICTIONARY, the privilege level is GLOBAL.

SYSTEM RELOAD DICTIONARY  →  GLOBAL

Correct grant

A global privilege must be granted globally:

GRANT SYSTEM RELOAD DICTIONARY
ON *.*
TO some_role;

Trying to replace *.* with analytics.* or events.* does not narrow the scope. It makes the grant invalid for this privilege.

Takeaway

Do not assume the ON database.* part of a ClickHouse GRANT can be used with every privilege. First check the privilege level.

If the privilege is GLOBAL, the scope is global too.

Problem

Our local Airflow setup used to install additional Python packages when the containers started, using runtime configuration such as:

_PIP_ADDITIONAL_REQUIREMENTS: >
  clickhouse-connect
  pytest
  pytest-mock
  freezegun

It was convenient, but startup now depended on package repositories, network access and dependency resolution. The image itself was no longer the complete description of the runtime.

The change

I moved the dependencies into a custom Airflow image:

FROM apache/airflow:3.0.6

COPY requirements.txt /requirements.txt

RUN pip install --no-cache-dir -r /requirements.txt

The important part was separating build from runtime:

build
  → install dependencies
  → immutable image

up
  → start existing containers

make up should start the environment. It should not silently build images, pull dependencies or install Python packages.

Production parity vs reproducibility

Keeping local development close to production is useful, but an exact copy is not always necessary. What matters more is that the runtime is predictable.

If two developers start the same image, they should get the same Python environment.

Result

Startup was not dramatically faster. The bigger improvement was that the environment stopped rebuilding itself every time it started.

Once the image existed locally, the stack could start without reaching package repositories or resolving dependencies again.

Takeaway

Runtime dependency installation is useful for experiments. For a development environment used every day, I prefer:

build once → run predictably

over:

start → download → install → hope

Reproducibility matters more than convenience.

Problem

An upstream task legitimately skipped when there was nothing to process. A downstream metrics task used trigger_rule="none_failed", so it was still allowed to run.

rows = load()
metrics(rows)

I initially handled the missing value inside the task:

rows = rows or 0

It did not help.

Why

The value passed between TaskFlow tasks is an XComArg. Before Airflow calls the Python function, it resolves that argument from XCom.

A skipped upstream task never pushed the return value, so resolution failed before the first line of metrics() executed.

Fix

Keep the dependency, but stop passing the upstream return value as a function argument:

load_result >> metrics()

Then inspect the upstream state explicitly and only read XCom when a return value should exist:

if load_state == "skipped":
    return

rows = ti.xcom_pull(
    task_ids="load",
    key="return_value",
) or 0

Takeaway

If an Airflow TaskFlow argument comes from XCom, handling None inside the task may already be too late.

Problem

A repository was created for every batch:

batch
  → repository
  → ClickHouse client
  → process
  → close

With around 74 batches, the pipeline created dozens of repository and client instances even though only five batches could run concurrently.

The resource lifetime was tied to the batch instead of the worker.

Why not share one client?

Using one repository for all threads was not safe either. The ClickHouse client maintained session state, so concurrent use from multiple threads could result in session-related errors.

The correct ownership boundary was:

one client per worker thread

Fix

Each worker keeps its own repository in thread-local storage and reuses it across batches.

before

74 batches
→ up to ~74 repositories / clients

after

5 workers
→ up to 5 reusable worker repositories

Clients were also made lazy, so a worker only opens a connection when it actually needs one. Worker resources are explicitly closed when processing finishes.

Result

Resource creation became bounded by worker count rather than batch count.

Load testing around the same path also exposed retained futures on error paths; fixing that lifecycle reduced observed peak memory growth from roughly 47 MiB to 9 MiB.

Takeaway

Limiting concurrency does not automatically limit resource creation.

The connection problem was not really a ClickHouse problem. It was an object-lifetime problem.