Multi-region system design looks impressive on a whiteboard. The difficult part begins when both regions are healthy enough to accept traffic but cannot agree on which one owns the data.

I used to approach multi-region system design questions with a fairly predictable architecture. Two regions, independent application deployments, regional databases, a global traffic router, and some form of replication between them. If the interviewer mentioned high availability, I would explain how traffic could move to the second region when the first became unavailable. The diagram looked professional, and most of the components were familiar enough that I could describe them without much preparation.

What I hadn’t spent enough time thinking about was what happened during the failover itself. Moving HTTP traffic between regions is relatively straightforward compared with deciding which database is allowed to accept writes. A region can become unreachable from one network while remaining fully operational from another, and a global load balancer cannot resolve that disagreement simply by choosing a different endpoint.

The problem became clearer when I started examining a payment platform with a requirement for regional resilience. The system needed to remain available during infrastructure failures, but it also needed to prevent duplicate payments and preserve account balances. Those requirements sounded compatible until I tried to explain how two regions would coordinate writes during a network partition.

I had been treating active-active deployment as an availability feature. In practice, it was also a consistency decision, and the consequences were much more expensive than the architecture diagram suggested.

The Multi-Region System Design Question Most Candidates Answer Too Quickly

Imagine a payment processing platform operating in two AWS regions. Customers connect through a global routing layer, and both regions run independent copies of the backend services. Each region has its own application instances, caches, message consumers, and database infrastructure.

The product team wants the platform to survive the loss of an entire region. They also want low latency for customers in different geographical locations, which makes active-active deployment attractive. From an infrastructure perspective, distributing traffic between regions appears to satisfy both requirements.

The difficulty is that payment processing is stateful. If a customer initiates a transfer in Region A and immediately checks the account balance through Region B, the second region needs a sufficiently current view of the transaction to produce a correct response. If both regions can independently approve payments against the same balance, asynchronous replication can allow them to make conflicting decisions.

Consider an account with $1,000 available. Region A receives a request to transfer $700 while Region B receives a separate request to transfer $600. If the regions have only asynchronously replicated state, both may observe the original balance and approve their transactions before learning about the other operation.

The combined transfers exceed the available balance. Each region processed its local request according to the information it had, but the global business invariant has been violated.

This is the first distinction I now establish in a multi-region system design discussion: application availability and write consistency are separate architectural requirements. The system can remain reachable while still being unable to safely authorize particular operations.

Active-Active Doesn’t Mean Every Database Should Accept Every Write

The phrase active-active is frequently used without enough precision. It can describe two regions serving read traffic, two regions handling independent customers, or two regions accepting concurrent writes to the same logical records. Those architectures have very different consistency requirements.

For a globally distributed content platform, concurrent updates may be manageable through versioning, conflict resolution, or data types designed for eventual convergence. If two users update different profile fields, the application may have a reasonable way to merge the changes. Some temporary inconsistency may also be acceptable.

A payment ledger has a different problem. Financial operations require clearly defined ownership, ordering, and invariants. Two conflicting balance updates cannot always be merged after the fact without changing the meaning of the transactions that were already approved.

For this reason, I would not begin a payment-system interview by promising unrestricted writes in both regions. I would first ask whether the platform requires active-active writes to the same account or whether different accounts can have designated write owners.

That distinction opens up several viable designs. An account might have a home region responsible for its authoritative writes, while other regions serve requests through routing or forwarding. Alternatively, the platform might use a distributed database that coordinates transactions across regions, accepting the latency and availability consequences of that choice.

The important part is explaining what each design guarantees when communication between regions becomes unreliable.

Why Asynchronous Replication Makes Failover Dangerous

Asynchronous replication is attractive because the primary region does not need to wait for a remote replica to acknowledge every write before responding. Under normal conditions, this can reduce write latency and allow the system to continue operating when a distant replica becomes temporarily unavailable.

The trade-off appears during failover.

Suppose Region A commits a payment transaction and returns success to the customer. The replication process has not yet delivered that transaction to Region B. A few milliseconds later, Region A becomes unavailable, and the routing layer directs new traffic to Region B.

Region B now has an incomplete history.

If it becomes the authoritative writer immediately, it may approve operations based on balances that do not include the last committed transactions from Region A. When Region A eventually returns, the system has to reconcile two histories that may contain conflicting financial decisions.

A simple architecture diagram rarely makes this risk visible.

Global Traffic Router
                         |
              +----------+----------+
              |                     |
           Region A              Region B
              |                     |
         Payment API           Payment API
              |                     |
         PostgreSQL A  ------> PostgreSQL B
                   Async Replication

The arrow between databases is doing a tremendous amount of work in this diagram. It represents replication lag, network reliability, failover behavior, and the question of which writes have become durable in the region that may take over.

When I discuss this design, I want to know the recovery point objective. If the business requires zero acknowledged-write loss, asynchronous replication without additional safeguards cannot simply be assumed to satisfy that requirement.

A system may choose to accept some potential data loss in exchange for lower latency and higher availability. That can be a valid decision for certain workloads, but it needs to be explicit.

The Split-Brain Failure That Changes the Entire Design

The most uncomfortable multi-region scenario isn’t always a complete regional outage. Sometimes both regions are still running, but communication between them has failed.

Region A continues accepting requests from clients that can reach it. Region B also appears healthy from its local perspective, and the global routing system may begin sending traffic there because health checks against Region A are failing from certain locations.

If both regions believe they are the primary writer, the system has a split-brain problem.

Each region can commit locally valid transactions while producing incompatible global state. The longer this continues, the more difficult reconciliation becomes, particularly when operations involve external systems such as payment providers.

A common mistake is to assume that health checks can reliably identify which region should be authoritative. Health checks can tell us whether a service is reachable from a particular vantage point. They cannot, by themselves, establish exclusive ownership of a distributed resource.

The architecture needs a stronger coordination mechanism.

One approach is to use a consensus-backed control plane that grants exclusive write authority to a particular region or shard. Another is to rely on a distributed database whose consistency model already incorporates the necessary coordination.

Either way, the design must account for the possibility that an old writer continues running after losing authority.

That is where fencing becomes important.

Fencing Tokens: Preventing an Old Primary From Continuing to Write

A lease-based leadership system may grant write authority to Region A for a limited period. If Region A becomes unreachable, the coordination system can eventually grant authority to Region B.

The difficulty is that Region A may not know it has lost leadership. A paused process, delayed network packet, or isolated service instance can continue attempting operations after its lease is no longer valid.

A fencing token provides a monotonically increasing generation number associated with leadership.

For example, Region A might hold generation 41. After failover, Region B receives generation 42. Every protected write must carry the current generation, and the authoritative storage or resource layer must reject operations using older generations.

This final enforcement point is essential. A fencing token checked only inside the application does not provide protection if the old leader can bypass that check and write directly to the resource.

The design also depends on how the current generation is established and validated. A consensus-backed authority can issue monotonically increasing generations, but the protected resource must enforce their ordering consistently.

For a database-backed workflow, this may require routing all writes through a single authoritative storage layer or implementing a transactionally enforced ownership protocol. A token attached to a request is not sufficient if two independent databases can each accept conflicting writes without coordination.

I would rather explain that limitation clearly in an interview than claim that adding a distributed lock solves split-brain.

Why Distributed Locks Are More Complicated Than They Look

Distributed locks are often proposed as a quick solution to regional ownership. The idea is that whichever region acquires the lock becomes the writer, and the other region waits.

That sounds reasonable until the lock holder loses connectivity to the lock service but remains connected to the database. The lock can expire and be acquired elsewhere while the original process continues executing work it started earlier.

A time-limited lease does not automatically revoke operations already in progress.

This matters when a service performs an external payment call or submits a database transaction after losing ownership. The system must ensure that stale operations cannot produce effects that conflict with the new leader’s decisions.

A reliable design therefore needs to connect leadership authority to the actual write path. Fencing, transaction validation, and carefully controlled ownership transitions are possible techniques, depending on the underlying infrastructure.

The external payment provider introduces another boundary. Even if the database rejects a stale write, an external charge may already have been initiated. Stable idempotency keys and reconciliation remain necessary because a distributed lock cannot make an arbitrary remote API participate in the same atomic transaction.

This is one reason financial systems are difficult to make fully active-active across regions. The business workflow crosses several independently failing systems, and no single coordination primitive automatically controls all of them.

The Latency Cost of Strong Consistency

A multi-region design that requires coordination across distant regions must account for network latency.

If a transaction needs acknowledgments from a quorum spanning geographically separated locations, the commit path includes network communication that may take tens or hundreds of milliseconds depending on placement and conditions. The exact cost depends on the database, replication protocol, quorum topology, and which replicas participate in the operation.

For a service targeting very low write latency, this can become a significant architectural constraint.

The candidate must decide whether the business needs globally coordinated writes for every operation or whether the data can be partitioned into independently owned groups.

Account-based ownership is one possibility.

If each account has an authoritative home region, operations involving that account can be routed to the appropriate writer. The platform can still operate in multiple regions, but it avoids allowing arbitrary regions to make conflicting decisions about the same account.

Cross-account transfers introduce additional complexity, especially when the accounts have different home regions. The design must then address how ownership, atomicity, and failure recovery work across those boundaries.

The advantage is that the architecture can make consistency requirements explicit rather than paying the cost of global coordination for every operation without considering whether it is necessary.

A More Defensible Architecture for a Multi-Region Payment Platform

For an interview scenario requiring regional resilience and strict financial invariants, I would begin with a simpler ownership model.

Each account or account group would have one authoritative write owner at a time. Requests could enter through either region, but operations affecting the same account would be routed to its current owner.

The authoritative region would record payment intent durably before initiating external side effects. Provider calls would use stable idempotency identifiers where supported, and uncertain outcomes would be handled through reconciliation rather than blind retries.

The system would replicate data to the secondary region according to a documented recovery objective. A failover controller would determine when ownership could transfer, and the write path would enforce the new ownership generation so that stale leaders could not continue committing protected state.

A conceptual design might look like this:

Global API Entry
                        |
                Account Routing
                        |
              Ownership Metadata
                        |
           +------------+------------+
           |                         |
       Region A                  Region B
           |                         |
     Payment Service           Payment Service
           |                         |
     Authoritative DB          Replica / Other
     for Owned Accounts        Account Owners
           |
     Durable Payment State
           |
     External Provider
           |
     Reconciliation

This is deliberately a conceptual architecture rather than a claim that a specific PostgreSQL replication setup automatically supports all of these guarantees.

The actual implementation depends on the database technology, failover mechanism, ownership protocol, and recovery requirements. In particular, the system must define what happens to writes that were acknowledged before a regional failure but were not replicated to the new owner.

If the business requires zero acknowledged-write loss, the design needs a durability protocol capable of supporting that guarantee. If it permits a nonzero recovery point objective, the application must have a documented process for detecting and reconciling missing operations.

Those decisions are more important than whether the diagram contains two regions or five.

The Recovery Procedure Is Part of the Architecture

I used to think of disaster recovery as an operational concern that could be addressed after the main system design was complete. That view misses an important fact: the recovery procedure determines whether the architecture’s consistency guarantees survive a failure.

Suppose Region A disappears.

The operations team needs to know whether Region B has received all acknowledged transactions, whether Region A might still be processing requests, and how the system will establish exclusive ownership before reopening writes.

Promoting a replica is only one step.

The application may also need to pause payment initiation, reconcile in-flight provider operations, update ownership metadata, invalidate stale routing decisions, and verify that the new writer has the required state.

When Region A returns, it cannot simply resume accepting writes as if nothing happened. Its database may contain a divergent history or transactions that were never replicated to Region B.

The recovered region must rejoin under a controlled process, potentially involving rebuilding replicas or reconciling durable business records.

A Staff Engineer should be able to explain that process at a high level. It reveals whether the architecture was designed for actual recovery or merely for an attractive availability diagram.

How I Practice Multi-Region System Design Questions Now

I no longer begin multi-region exercises by deciding how many regions to deploy.

I start with the data.

Which operations require strict ordering? Which records can tolerate stale reads? What is the acceptable loss of acknowledged writes, and how quickly must the platform recover after a regional outage?

Once those requirements are clear, I introduce the failure cases that are most likely to invalidate the design.

I consider a complete region outage, a network partition between regions, delayed replication, a stale primary continuing to process requests, and a failover that occurs while external side effects are in progress.

Then I examine whether the proposed ownership and replication mechanisms preserve the required business invariants.

This approach is slower than drawing two regional clusters and connecting them with arrows. It also produces a much more useful architecture discussion because every component has a reason to exist.

When I built my own system design preparation material, these were the types of questions I wanted to practice. Understanding a pattern is useful, but the difficult part is explaining its behavior when the assumptions behind it stop holding.

That preparation became the Senior & Staff System Design Interview OS, a 130-page resource with an 89-page interview playbook and 36 architecture diagrams in a separate Visual Architecture Atlas. I built it from the methods I used in my own preparation, with an emphasis on architecture trade-offs, failure recovery, capacity decisions, and the follow-up questions that expose weaknesses in a design.

What Changes at Staff Engineer Level?

A Senior Engineer should be able to explain how a multi-region service behaves during normal traffic and common infrastructure failures. The discussion should include routing, database replication, consistency, monitoring, and recovery.

At Staff level, I would expect more attention to the cost and feasibility of the architecture.

For example, an active-active design may require changes to the application’s data model, operational processes, and team ownership boundaries. A migration from single-region writes to distributed ownership may need to happen incrementally while the existing payment platform continues operating.

The candidate should consider how to test failover without risking financial correctness, how to measure replication lag, and how to verify that stale writers cannot commit transactions after leadership changes.

There are also business trade-offs.

A platform that must remain writable during every possible regional partition may be incompatible with strict consistency for shared financial state. The candidate needs to explain which requirement takes priority and how the product behaves when both cannot be satisfied simultaneously.

That conversation is much closer to the decisions senior technical leaders actually make.

Frequently Asked Questions About Multi-Region System Design

What is active-active architecture in system design?

Active-active architecture generally means multiple deployments actively serve production traffic. It does not necessarily mean every region can write to the same data simultaneously. The precise consistency and ownership model depends on the application and database design.

What is the difference between active-active and active-passive architecture?

In active-passive designs, one deployment typically handles authoritative operations while another remains available for failover. Active-active designs allow multiple deployments to serve traffic concurrently, although write ownership may still be partitioned or restricted. The right choice depends on availability, latency, and consistency requirements.

What is split-brain in distributed systems?

Split-brain occurs when multiple nodes or regions independently believe they have authority over the same resource. Without effective coordination and enforcement, they may accept conflicting writes. Leadership protocols and fencing mechanisms can help prevent stale owners from continuing protected operations.

How do you prevent split-brain during regional failover?

A robust approach uses an authoritative coordination mechanism to establish exclusive ownership and enforces that ownership at the protected resource. Fencing tokens can prevent stale leaders from committing writes when the resource validates generations correctly. The design must also handle network partitions and incomplete operations.

Can PostgreSQL support multi-region active-active writes?

Standard PostgreSQL streaming replication uses a primary-replica model rather than unrestricted multi-primary writes. Multi-region write architectures require additional technology or application-level ownership and conflict-handling mechanisms. The guarantees depend on the chosen implementation.

What happens to unreplicated data during failover?

If a primary acknowledges a write before an asynchronous replica receives it, promoting that replica can result in the acknowledged write being absent from the new primary. Applications must account for this through recovery objectives, replication choices, and reconciliation procedures.

Does synchronous replication guarantee zero data loss?

Synchronous replication can strengthen durability guarantees, but the exact behavior depends on which replicas acknowledge writes, how failover is performed, and the database configuration. It should not be treated as a universal zero-data-loss guarantee without examining the full protocol and failure model.

Why is active-active difficult for payment systems?

Payment systems often require strict invariants, ordered state transitions, and coordination with external providers. Concurrent writes in separate regions can create conflicting decisions if replication is asynchronous. Preventing those conflicts requires an appropriate ownership or consistency model.

What is a fencing token in distributed systems?

A fencing token is a monotonically increasing value associated with a leadership generation or resource ownership grant. A protected resource can reject operations carrying older generations, preventing stale leaders from continuing to modify state. The token is effective only when enforcement occurs at the relevant resource boundary.

What should a Staff Engineer discuss in a multi-region system design interview?

A Staff Engineer should address consistency requirements, failure domains, data ownership, replication behavior, failover procedures, operational complexity, and migration strategy. The discussion should also cover how the architecture’s guarantees are tested and how teams respond when a region returns after an outage.

The Diagram Became Simpler Once I Understood the Failure

The architecture I would draw today is less impressive at first glance than the one I used to prefer. It may have a single authoritative writer for each account, fewer cross-region dependencies, and an explicit recovery process that temporarily restricts certain operations during a partition.

That is a deliberate trade-off.

A system handling financial transactions needs to preserve its business invariants even when the infrastructure behaves unpredictably. Deploying more regions can improve resilience, but it also creates additional ways for state to diverge if ownership and consistency are poorly defined.

I still consider active-active architecture when the requirements justify it. I simply want to understand which operations can safely run concurrently, how ownership is established, and what the platform will do when regions disagree.

Those questions make the design harder to explain, but they also make it much more credible.

More Advanced System Design and Distributed Systems

I publish deeper engineering breakdowns on my free Substack, covering distributed systems, production failures, architecture decisions, and the problems that emerge when real infrastructure behaves differently from a clean diagram.

I also created the Senior & Staff System Design Interview OS from my own preparation experience. It includes 130 pages, 36 architecture diagrams, and an 89-page playbook focused on requirements, architecture trade-offs, scaling, failure recovery, and challenging interviewer follow-up questions.

The material is intended for experienced engineers who want to practice defending architectural decisions under changing requirements rather than relying on memorized diagrams. You can explore the complete System Design Interview OS here.

active-active architecture, multi-region failover, split-brain distributed systems, database replication consistency, Staff Engineer system design interview, distributed systems fencing tokens, multi-region payment architecture.

I Thought Active-Active Architecture Was the Best Answer. Then I Had to Explain Failover was originally published in Javarevisited on Medium, where people are continuing the conversation by highlighting and responding to this story.

Our PostgreSQL queries were expensive, the cache hit rate looked excellent, and the dashboard got faster. The incident started when a customer changed something that Redis still believed was true.

The performance problem was straightforward enough. Our SaaS dashboard had become increasingly expensive to render as customers accumulated more data. PostgreSQL was handling repeated aggregation queries, several API endpoints were requesting the same account information, and response times deteriorated during peak traffic. Redis looked like the obvious solution because much of the information was read frequently and changed relatively infrequently.

The first benchmark was encouraging. We cached account summaries, reduced repeated database queries, and brought the dashboard’s p95 latency down from roughly 1.8 seconds to 240 milliseconds in our test environment. The Redis hit rate exceeded 95%, PostgreSQL CPU pressure dropped, and the infrastructure graphs finally looked the way we’d wanted them to look for months.

Then a customer upgraded their subscription and immediately tried to use a feature included in the new plan. The payment had succeeded, the subscription record was correct in PostgreSQL, and the billing service had published the expected update event. The API still rejected the request because its cached account entitlements said the customer was on the previous plan.

The customer didn’t care that our cache hit rate was excellent. From their perspective, we’d taken their money and failed to deliver the product.

This is a reconstructed engineering case with illustrative measurements, based on real failure modes in Redis-backed SaaS architectures. It is intended to show how I’d investigate and redesign the system, rather than claim results from an identifiable customer engagement.

The First Version Used Cache-Aside Everywhere

Our original implementation followed the cache-aside pattern. The application checked Redis before querying PostgreSQL, loaded the data on a miss, and cached the result for subsequent requests.

The code was easy to understand:

public AccountSummary getAccountSummary(UUID accountId) {
    String key = "account:summary:" + accountId;

  AccountSummary cached = redis.get(key, AccountSummary.class);
    if (cached != null) {
        return cached;
    }
    AccountSummary summary = repository.loadSummary(accountId);
    redis.set(key, summary, Duration.ofMinutes(15));
    return summary;
}

For read-heavy account summaries, this was a reasonable starting point. The cache reduced database work and improved response times when the underlying data hadn’t changed.

We also reused the account summary in other services. That decision mattered more than the initial performance improvement.

The same cached object contained dashboard information and subscription entitlements. Some fields were suitable for eventual consistency, while others controlled whether the customer could access paid functionality.

We had assigned one cache expiration policy to data with different correctness requirements.

A fifteen-minute delay in a usage chart might be acceptable. A fifteen-minute delay in recognizing a paid upgrade was a product failure.

The implementation treated both cases as ordinary stale-cache behavior.

Our Cache Invalidation Worked Until Two Requests Arrived Together

When a subscription changed, the billing service updated PostgreSQL and deleted the corresponding Redis key.

The assumption was that the next read would fetch the updated subscription from the database and repopulate the cache.

That sounds correct until a read overlaps the write.

Consider this sequence:

1. Request A misses the cache and begins reading the old subscription state from PostgreSQL.
2. Request B commits the subscription upgrade and invalidates the Redis key.
3. Request A finishes its earlier database read and writes the old state back into Redis.
4. Subsequent requests receive the stale value until another invalidation or expiration.

The invalidation itself succeeded. The race happened because the cache fill wasn’t coordinated with the concurrent database update.

This is one reason cache-aside invalidation is harder than it appears in architecture diagrams.

A cache delete followed by a database read doesn’t guarantee that the resulting cache entry represents the latest committed state. Another request can be operating on an earlier snapshot or on data retrieved before the update completed.

We initially shortened the TTL from fifteen minutes to sixty seconds.

That reduced the maximum lifetime of some stale entries, assuming the entry wasn’t continually refreshed, but it didn’t remove the race.

For an entitlement check, sixty seconds of incorrect authorization behavior could still be unacceptable.

We needed to stop treating expiration time as a substitute for a consistency design.

The Redis Hit Rate Was Hiding the Problem

The infrastructure dashboard reported a 96% cache hit rate.

That number had become one of our favorite performance metrics because it correlated with lower database load. Unfortunately, it said nothing about whether the cached data was correct.

A stale value returned in one millisecond still counts as a successful cache hit.

The more often an incorrect value is served, the better the cache hit-rate graph can look.

We began measuring something different: how frequently cached versions disagreed with authoritative database versions during controlled validation.

For the account summary, we introduced a version field that changed whenever relevant source data changed.

The cached object contained the version alongside its payload. This gave us a way to reason about cache freshness instead of relying entirely on elapsed time.

Versioning doesn’t automatically make a cache consistent. The application still needs a trustworthy way to determine whether a version is current, and checking PostgreSQL on every read can defeat the performance benefit.

But version information becomes useful when coordinating writes, invalidation events, and conditional cache updates.

It also makes stale-state investigations far easier. Instead of asking whether Redis might contain old data, engineers can compare the cached version with the version associated with a committed business update.

That was a much better starting point for debugging.

We Had Mixed Presentation Data With Authorization Decisions

The architecture review exposed a more serious mistake.

Our account summary object was serving two different purposes.

The dashboard used it to display subscription information, recent usage, and account metadata. The authorization layer used part of the same object to decide whether a customer could access a paid feature.

Those operations have different failure consequences.

If a dashboard displays a subscription change a few seconds late, the experience may be mildly confusing. If an authorization decision uses outdated entitlements, the application can deny a legitimate customer or, depending on the direction of the change, continue granting access after permissions have been revoked.

The second case can become a security problem.

We separated the data models.

The dashboard continued using Redis for information that could tolerate bounded staleness. Entitlement checks moved to a more carefully controlled path with explicit freshness requirements.

That didn’t mean querying PostgreSQL for every authorization decision in every service. There are valid designs for caching authorization data, including versioned policies and short-lived authorization state.

The important change was making the consistency contract explicit.

For each operation, we needed to know whether stale data was acceptable, for how long, and what the application should do when it couldn’t establish freshness.

A cache should never determine those requirements accidentally.

The Next Problem Was a Cache Stampede

Once we corrected the entitlement behavior, another performance issue appeared during load testing.

A frequently requested account summary expired while a large number of requests were arriving.

Several application instances observed the cache miss at nearly the same time. Each instance queried PostgreSQL and attempted to rebuild the same expensive summary.

Redis had reduced database load during normal operation, but a synchronized miss sent a burst of identical work back to PostgreSQL.

The application had a cache stampede.

This became particularly visible for large tenants. Their summaries required more aggregation work, and popular accounts could generate significant concurrent demand.

We evaluated request coalescing so that multiple requests for the same key wouldn’t independently execute the same database query.

Within a single application instance, a local single-flight mechanism can ensure that one request performs the load while others await its result.

Across multiple instances, coordination becomes more complicated.

A distributed lock can reduce duplicate cache fills, but it introduces its own failure behavior. Locks expire, processes crash, and a worker can continue executing after its lease has expired.

If that worker eventually writes an old value into Redis, it may overwrite a newer result produced by another instance.

The solution needs to consider stale writers, not merely concurrent cache misses.

For some workloads, serving a slightly stale value while one worker refreshes it is a reasonable trade-off. For others, particularly authorization and financial state, that behavior may be unacceptable.

The correct strategy depends on the business operation.

We Used Version Checks to Prevent Old Workers From Overwriting New Data

For cache entries that required stronger protection against stale writes, we evaluated conditional updates based on monotonically increasing versions.

The principle was simple.

A cache fill carrying version 41 should not overwrite an existing cache entry at version 42.

Redis supports atomic server-side operations, so the version comparison and update can be implemented as one atomic action, for example through a Lua script or another appropriate Redis mechanism.

Conceptually, the update follows this rule:

incoming.version > cached.version
    -> replace cached value

incoming.version <= cached.version
    -> keep existing value

That rule is useful, but it requires a reliable version source.

A timestamp generated independently by different application instances is a poor choice when clock differences or concurrent updates can affect ordering.

We preferred versions derived from authoritative database state, with increments controlled by the transaction that changed the underlying business record.

Even then, versioned cache updates don’t eliminate every consistency problem.

If the cache is empty, a delayed reader may still populate it with an older version unless the system retains or can verify a more recent version boundary. Invalidation events can also arrive out of order.

For stronger guarantees, the design may need a version watermark, tombstone, authoritative version check, or a different read path altogether.

We deliberately avoided claiming that one Lua script made Redis strongly consistent with PostgreSQL.

It didn’t.

It gave us a mechanism for preventing a specific class of stale overwrite, provided the surrounding protocol maintained the necessary version information.

The Database Commit and the Invalidation Event Were Another Failure Boundary

Our subscription service updated PostgreSQL and then published an event to notify other services that account state had changed.

That created a familiar dual-write problem.

If PostgreSQL committed successfully but the application crashed before publishing the event, downstream caches might never receive the invalidation.

The TTL would eventually remove the stale entry, but the system had no reliable guarantee about how quickly downstream services would learn about the committed change.

We considered using a transactional outbox.

The service would write the subscription update and an outbox record in the same PostgreSQL transaction. A separate publisher would deliver the event to the messaging system.

This improved the reliability of recording the intent to publish.

It didn’t guarantee that every consumer would receive exactly one notification. Events could be delivered more than once, and consumers still needed to handle duplicate and potentially out-of-order notifications.

For cache invalidation, duplicate delivery is often harmless when the operation is designed to be idempotent.

Out-of-order events require more thought, especially when consumers update cached values rather than simply deleting them.

We chose to make the event carry the business entity identifier and its committed version. That gave downstream consumers enough information to avoid applying older state over newer state.

The event was a signal that authoritative data had changed, rather than an invitation to trust an arbitrary cached snapshot.

This reduced the coupling between event delivery timing and the correctness of the read model.

We Stopped Treating Redis as a Faster PostgreSQL

The architecture had gradually drifted into a dangerous assumption: if Redis contained a value, the application could use it as though it were authoritative.

That was never the contract.

PostgreSQL owned the subscription state. Redis held a derived representation optimized for selected reads.

Once we made that distinction explicit, several decisions became easier.

We could cache dashboard summaries aggressively because their freshness requirements were relatively relaxed. We could treat entitlement changes as consistency-sensitive operations and use a stricter validation path. We could allow stale-while-revalidate behavior for selected analytics data without extending the same policy to billing or authorization.

We also changed our monitoring.

Cache hit rate remained useful, but we added metrics for cache fill duration, concurrent misses, invalidation failures, stale-version rejections, and the age of cached data where that could be measured reliably.

For critical workflows, we added tests that exercised overlapping reads and writes.

A typical integration test would deliberately pause a cache fill after reading an old database version, commit a newer version through another transaction, and then allow the original cache fill to continue.

The test would verify that the old result could not replace the newer cache state under the chosen protocol.

This was much more valuable than another unit test proving that redis.get() returned a value.

The Performance Results Became Less Impressive and More Trustworthy

After the redesign, we repeated the load test with realistic tenant sizes and concurrent subscription updates.

We measured latency and database load, but we also measured correctness under overlapping operations.

The illustrative results looked like this:

Metric                         Before Redis   Initial Cache   Redesigned
----------------------------------------------------------------------------
Dashboard p95 latency          1,800 ms       240 ms          310 ms
PostgreSQL queries / request   18             2               3
Cache hit rate                 N/A            96%             93%
Observed stale entitlement
decisions in test scenarios    N/A            Present         None observed
Concurrent cache-fill bursts   N/A            Frequent        Reduced

These are example benchmark figures, not results from a verified production engagement. The table is intended to show the trade-off we’d want to evaluate, including correctness alongside performance.

The redesigned endpoint was slightly slower than the first cached implementation.

That was acceptable.

The initial version had achieved impressive latency by allowing a cache entry to influence business decisions without sufficient freshness controls.

The revised architecture preserved most of the performance benefit while separating consistency-sensitive operations from ordinary presentation data.

We didn’t optimize for the highest possible Redis hit rate. We optimized for a system that could serve customers quickly without making incorrect subscription decisions.

What I’d Review Before Adding Redis to Another SaaS

I would start by classifying the data rather than choosing a TTL.

Which values are derived summaries? Which represent authoritative business state? Which decisions become dangerous if the data is stale?

Then I’d map the write path.

A PostgreSQL transaction commits, but how do the caches learn about that change? What happens if the process crashes between commit and notification? Can invalidation events arrive out of order, and can an older reader repopulate a key after a newer write?

I’d examine cache-fill concurrency and expiration behavior under realistic load. A cache that performs well with warm keys may behave very differently during a cold start, Redis failover, or a synchronized expiration event.

I’d also inspect the operational consequences of Redis being unavailable.

Does the application fall back to PostgreSQL? If every instance falls back simultaneously, can the database absorb the traffic? Are some operations allowed to fail closed, and can the system distinguish those from ordinary dashboard reads?

Those questions belong in the architecture review before the cache becomes a critical production dependency.

Redis can be an excellent performance tool. It becomes dangerous when teams let its speed obscure the consistency guarantees their business actually needs.

Our Fastest Version Wasn’t the One We Kept

The original cache implementation improved the benchmark dramatically. It also introduced a failure mode that our performance dashboard couldn’t detect.

A customer could complete a payment, receive confirmation, and still be denied access because Redis held an outdated view of the account.

The database was correct. The cache was fast. The product behavior was wrong.

We kept Redis because it was useful for the workload. We changed the boundaries around what it was allowed to decide, how cached state was updated, and which failures the application had to tolerate.

The result was an architecture with more explicit consistency rules and fewer assumptions hidden inside convenience methods.

For a SaaS engineering team, that is the difference between making an endpoint faster and making the system better.

Backend Architecture & Production Reliability Consulting

I’m Devrim Ozcay, founder of ProdRescue AI, an independent engineering consulting practice focused on Java and Spring Boot systems, PostgreSQL, Redis, distributed architectures, and production reliability.

I work with SaaS founders, CTOs, and engineering leaders who need an external technical assessment of caching strategies, data consistency risks, transaction boundaries, performance bottlenecks, and recurring production incidents.

Our engagements include architecture reviews, technical risk assessments, prioritized recommendations, and ongoing engineering advisory. Monthly consulting plans start at $750.

Explore consulting services: https://devrimozcay.gumroad.com/l/uwzfw

ProdRescue AI website: https://www.prodrescueai.com/

Business inquiries: info@prodrescueai.com

If your team has added Redis but still struggles with stale data, inconsistent authorization, cache stampedes, or unpredictable database load, email me a brief description of the architecture and the behavior you’re observing. We can discuss whether a focused technical review would help identify the underlying risks.

Frequently Asked Questions

Why does Redis return stale data after PostgreSQL is updated?

Cache-aside systems can experience races between database writes, invalidation, and concurrent cache fills. A reader may repopulate Redis with an older value after another request has committed a newer database state.

How do you prevent cache stampedes in Redis?

Common approaches include request coalescing, controlled refresh, staggered expiration, and carefully designed distributed coordination. The correct choice depends on whether the data can tolerate stale reads and how cache failures affect the database.

Is Redis safe for caching authorization and subscription entitlements?

It can be used with an explicit consistency and freshness design. Sensitive authorization decisions require stronger controls than ordinary dashboard data, especially when stale values could grant or deny access incorrectly.

Does deleting a Redis key after a database update guarantee consistency?

No. Concurrent readers can repopulate the key using data retrieved before the update. Additional coordination, versioning, or authoritative validation may be necessary depending on the required guarantees.

How do you prevent old cache writers from overwriting newer values?

Version-aware conditional updates can reject older values when the cache already knows a newer version. Stronger protocols may also require version watermarks or authoritative checks to handle empty keys and delayed readers.

When should a SaaS company conduct a Redis architecture review?

A review is useful when cached data influences billing or authorization, stale reads affect customers, cache misses overload PostgreSQL, or the team cannot explain how invalidation behaves during concurrent updates and failures.

Redis cache invalidation, Redis stale data PostgreSQL, Spring Boot Redis caching, Redis cache stampede, cache aside race condition, Redis PostgreSQL consistency, Redis distributed cache architecture, SaaS backend performance.

We Added Redis to Fix a Slow SaaS Dashboard. Then Customers Started Seeing the Wrong Data was originally published in Engineering Under Pressure on Medium, where people are continuing the conversation by highlighting and responding to this story.

I was one conversation away from a $185,000 offer. The final interview was supposed to be an informal meeting with the team. I hadn’t expected a developer with eighteen months of experience to expose the biggest weakness in my engineering career.

The junior developer had been quiet for most of the interview. His camera was on, but he hadn’t said much beyond introducing himself and explaining that he worked on the company’s billing platform. I assumed he was there to meet a potential teammate rather than evaluate my technical skills.

About thirty minutes into the call, he interrupted the Engineering Manager and asked whether he could show me something.

His name was Ethan. He shared a small Java method from their codebase and explained that he’d written it a few weeks earlier. It wasn’t particularly complicated, just a Spring Boot service method responsible for updating a customer’s subscription after a payment succeeded.

He wanted to know whether I’d approve the pull request.

I read through the code and immediately noticed that the method was doing too much. It validated the subscription, updated the database, sent an email, and published an event to another service. I explained that I’d separate those responsibilities and make the method easier to test.

Ethan listened carefully. Then he asked, “Could you show me what that would actually look like?”

I smiled because I thought it would be easy. Twenty minutes later, I was wishing I’d kept my mouth shut.

Six Interviews Had Made Me Overconfident

I’d been interviewing for Senior and Staff-level backend positions for nearly two months. My previous job involved maintaining Java services, improving database performance, and handling production incidents in a financial platform. I had enough experience to speak comfortably about distributed systems, architecture decisions, and the operational consequences of bad code.

The first six interviews had gone unusually well.

I’d passed two coding assessments, three system design rounds, and a behavioral interview with an Engineering Director. The company was building a subscription billing platform, and the role involved helping a team of twelve engineers improve its backend architecture.

The recruiter had mentioned a compensation package around $185,000, depending on the final leveling decision. That was more than I’d earned before. By the time I reached the team interview, I’d started imagining myself accepting the offer. I was already thinking about the projects I’d work on, the systems I’d improve, and how I’d introduce better engineering practices.

I hadn’t considered that someone might ask me to demonstrate those practices in front of the people who would actually have to live with them.

The Pull Request Looked Familiar

Ethan’s method was called activateSubscription.

It loaded a subscription from PostgreSQL, checked whether the payment had been completed, changed the subscription status, saved the entity, sent a confirmation email, and published a SubscriptionActivated event. The entire operation was wrapped in a Spring transaction.

I’d seen similar code in production dozens of times.

The obvious concern was that the method performed external operations while a database transaction was active. If the email provider responded slowly, the transaction could remain open longer than necessary. If the event publisher failed after the database changes were made, the behavior would depend on how the exception was handled and whether the transaction rolled back.

I explained those risks. The Engineering Manager nodded, and I could tell he agreed with my assessment. Then Ethan asked me to rewrite the method.

I started by extracting the email logic into a separate service. I moved the event publishing into another component and introduced an interface to make the implementation easier to test. Within ten minutes, I’d created several new classes and changed the original method into a sequence of smaller calls. It looked cleaner. At least, it looked cleaner to me.

Ethan asked what would happen if the database transaction committed successfully but the application crashed before publishing the event.

I told him we’d need to handle that failure. He asked how. I suggested retrying the event publication. He asked where the retry would get the event from if the process had already crashed.

I stopped typing.

The Junior Engineer Had Found the Problem I Was Avoiding

I’d been so focused on separating responsibilities that I hadn’t addressed the consistency problem.

Moving an external call into another class didn’t make it reliable. The application could still update the subscription without notifying downstream systems, and a process crash could leave no durable record of the event that needed to be published.

Ethan wasn’t being difficult.

He was asking the exact question that mattered.

I explained that we’d need to persist the event or an equivalent record as part of the same database transaction that updated the subscription. A separate worker could then publish it asynchronously and retry failures. We’d also need to consider duplicate delivery because publishing an event and marking it as delivered couldn’t generally be made atomic across the database and an external broker.

Ethan nodded. Then he asked whether that meant we should introduce an outbox table. I said it was one reasonable option. He asked what would happen if the event was published successfully but the worker crashed before marking it as delivered.

I explained that the event could be published again, so consumers would need to handle duplicates safely. Depending on the downstream operation, that might involve a stable event identifier and an idempotent consumer implementation. The Engineering Manager was watching us closely. I realized the interview had become much more technical than the system design rounds I’d already passed.

And I was struggling to keep up with a developer who’d been working professionally for less than two years.

Then Ethan Asked About the Email

We’d spent almost fifteen minutes discussing reliable event publication when Ethan returned to the confirmation email.

He asked whether the email should be sent before or after the subscription was committed. I said after. He asked what should happen if the email provider was unavailable for several hours. I suggested putting the notification into a queue. He asked whether we’d need a separate outbox mechanism for that too.

I said it depended on how the notification workflow was designed and what guarantees the business required. If confirmation emails were important enough to require reliable delivery, we’d need a durable record of the notification request. We also needed to consider whether sending the same email twice would be acceptable.

Ethan looked satisfied. Then he asked something I hadn’t expected.

“Would you actually build all of this for our current workload?”

I looked at the method again. The service handled a few thousand subscription activations per day. The company already had a background job system, and the existing billing platform contained a mechanism for retrying failed notifications.

I’d spent twenty minutes designing new infrastructure without asking whether the team already had something that solved the problem. I admitted that I hadn’t asked enough questions about their existing architecture. Ethan said they already used a transactional outbox for several payment-related workflows. The method he’d shown me was part of an unfinished pull request. He wanted to know whether I’d help him integrate it with the existing infrastructure rather than create another mechanism.

I felt my face getting warm.

The Engineering Manager Asked Me to Stop Coding

We had about ten minutes remaining when the Engineering Manager interrupted.

He said we’d covered enough technical ground and wanted to discuss how I’d work with the team. He asked what I’d do if Ethan submitted a pull request like this during a normal sprint. I gave the answer I thought a senior engineer should give. I’d explain the risks, suggest improvements, and help him understand the failure scenarios. The manager asked whether I’d rewrite the code myself. I said probably not. Then he pointed out that I’d spent most of the exercise doing exactly that.

He wasn’t wrong.

I’d taken over the screen, created several classes, and started restructuring Ethan’s implementation before understanding the constraints. I’d treated the exercise as an opportunity to demonstrate my technical ability rather than help another engineer improve his work.

The manager asked how I’d approach the review differently if we started again.

I thought about it for a moment.

I’d begin by asking Ethan what behavior the method needed to guarantee and what existing components were available. I’d ask which failure scenarios he’d already considered and what tests he’d written. Then we’d work through the remaining risks together instead of me immediately replacing his design with my own. That answer sounded much better. Unfortunately, it came after I’d spent half an hour demonstrating the opposite behavior.

I Didn’t Get the Offer

The recruiter called two days later. She said the team had decided to move forward with another candidate. My technical feedback had been strong, but the team interview had raised concerns about how I’d collaborate with less experienced engineers.

I asked whether she could share anything more specific.

She said the team felt I’d moved too quickly into solution mode and hadn’t spent enough time understanding Ethan’s approach. They wanted someone who could help engineers develop their own judgment rather than take over difficult problems.

I thanked her for the feedback.

Then I sat at my desk and thought about how absurd the situation was. I’d passed six interviews covering Java, system design, databases, and engineering leadership. I’d discussed distributed transactions, idempotency, concurrency, and reliability with experienced engineers. And I’d lost the offer during a conversation with a junior developer.

The worst part was that I couldn’t honestly say the decision was unfair.

I Started Thinking About My Previous Team

At my last company, junior engineers often came to me when they were stuck.

I’d usually open their code, identify the problem, and show them how I’d solve it. Sometimes I’d write a quick example or refactor a method while explaining my reasoning. I thought I was being helpful because I could resolve their issues quickly.

I rarely asked how they’d approached the problem before coming to me.

I also didn’t always give them enough time to work through the solution themselves.

One engineer used to ask me for help with almost every database-related issue. I assumed he lacked confidence with SQL and transaction handling. Looking back, I wondered whether I’d trained him to depend on my answers because I kept solving problems before he had a chance to understand them.

I remembered a code review where he’d proposed a reasonable solution using an existing repository method. I’d replaced it with a more elaborate implementation because I thought it was cleaner. The code worked. But I’d never explained why the additional complexity was necessary. Maybe it wasn’t.

The Next Interview Was Different

Three weeks later, I interviewed for another Senior Backend Engineer position.

The company asked me to review a pull request involving a scheduled job that processed failed payments. The implementation had several problems, including a retry mechanism that could repeatedly process the same payment without checking whether the previous attempt had succeeded.

This time, I didn’t immediately start rewriting the code.

I asked the interviewer to explain the intended behavior and the guarantees provided by the payment provider. We discussed how the application recorded attempts, what happened when a request timed out, and whether retries could create duplicate charges.

Then I described the risks I’d identified.

I explained how I’d discuss them with the engineer who wrote the code and what tests I’d want before approving the change. We worked through a safer implementation together, including how to represent uncertain payment outcomes and prevent repeated business effects.

The interviewer asked whether I’d prefer to implement the fix myself.

I said that depended on the urgency and the engineer’s experience. If customers were actively being charged incorrectly, I’d prioritize resolving the incident. During ordinary development, I’d rather help the engineer understand the problem and implement the change with appropriate support. The conversation felt slower than my previous interviews. I was also much more comfortable with it. The company made an offer the following week.

What That Junior Engineer Actually Taught Me

I used to think seniority meant being able to solve difficult problems faster than everyone else.

That belief had helped me develop strong technical skills, but it had also made me impatient. I was quick to recognize familiar patterns and sometimes too eager to demonstrate that I knew the solution. When another engineer struggled, I’d often treat the situation as a technical problem that needed to be resolved immediately.

Ethan showed me the limitation of that approach.

He didn’t need someone to replace his code with a more elaborate implementation. He needed someone who would understand the system he was working in, challenge the assumptions that mattered, and help him make a decision he could maintain afterward. I’d missed that distinction during the interview.

The experience also changed how I think about code reviews.

A good review should improve the software, but it should also leave the engineer who wrote it with a clearer understanding of the problem. Sometimes that means explaining a concurrency issue or identifying a missing failure scenario. Other times it means asking a question and giving someone enough room to work through the answer.

I still believe experienced engineers should be decisive when production systems are at risk. There are situations where taking control is necessary, especially during incidents with immediate customer impact. But ordinary engineering work offers opportunities to build judgment across a team, and those opportunities disappear when one person insists on providing every answer.

I lost a $185,000 opportunity because I treated a junior developer’s pull request like a coding exercise. The technical problems I identified were real. My mistake was assuming that identifying them made me the most useful person in the conversation.

Prepare for Senior and Staff Engineering Interviews

If you’re preparing for advanced backend interviews, my Senior Java & Spring Boot Engineering OS covers practical Java, Spring Boot, concurrency, database transactions, and production engineering problems. For architecture-focused interviews, the Senior & Staff System Design Interview OS helps you practice the technical trade-offs and failure scenarios experienced engineers are expected to explain.

You can also start with my Free Senior Engineer Interview & Production Playbook. I publish more engineering career stories, technical discussions, and lessons from production systems on Substack, where I write about what separates experienced engineers from people who simply know the technology.

I’m building ProdRescue AI System Design Lab, a platform for practicing realistic Senior and Staff Engineer system design interviews with structured feedback. If you’d like to help shape the product and be considered for early access, join the private beta research here.

I Passed 6 Senior Engineer Interviews. Then a Junior Developer Asked Me to Explain My Code was originally published in Engineering Under Pressure on Medium, where people are continuing the conversation by highlighting and responding to this story.

I had spent two months preparing for modern backend interviews. The company that finally made me an offer wanted to know whether I could work on a system nobody dared to upgrade.

The interviewer shared his screen and opened a Java class that was almost 1,400 lines long. There were SQL queries embedded directly in service methods, exception handlers that swallowed errors, and a comment from 2017 warning future developers against changing a particular condition. The application was still running in production, processing thousands of business transactions every day.

He asked me what I’d do with it.

I laughed because I thought he was joking. I’d spent the previous six weeks studying distributed systems, Kubernetes, Kafka, and every complicated system design question I could find. Now a company was showing me a piece of software that looked like it had survived several engineering teams and at least one organizational restructuring. I said I’d probably start by writing tests. He nodded and asked which tests. That was when I realized this interview would be different.

Seven Interviews Had Made Me Question My Experience

I’d been a Java backend engineer for nearly eight years. My work involved Spring Boot services, PostgreSQL databases, payment processing workflows, and the occasional production incident that turned an ordinary Tuesday into a very long evening. I wasn’t a famous open-source contributor, and I hadn’t designed infrastructure for hundreds of millions of users, but I’d spent enough time maintaining business-critical software to know what happened when seemingly harmless changes reached production.

The job search had started badly.

The first company rejected me after a coding exercise involving a graph traversal problem. I understood the algorithm but made an implementation mistake under pressure, and the interviewer didn’t seem particularly interested in discussing how I’d debug it. The second company wanted extensive experience operating Kubernetes clusters, even though the role was advertised primarily as backend development.

The third interview involved designing a real-time analytics platform. I spent an hour discussing event ingestion, partitioning, and data processing, then received feedback saying my design hadn’t demonstrated sufficient experience with large-scale infrastructure. I could understand the criticism, but it was difficult to reconcile with the work described in the job posting.

By the fifth interview, my preparation routine had become ridiculous.

I’d wake up early to solve coding problems, spend lunch reviewing system design diagrams, and study Java concurrency after dinner. My notes contained explanations of the Java Memory Model, database isolation levels, Kafka delivery guarantees, and enough distributed systems terminology to make an architecture meeting unbearable. I was getting better at answering interview questions. I wasn’t getting any closer to an offer.

The Seventh Rejection Was Particularly Embarrassing

The company asked me to design a service that could handle millions of scheduled tasks. I proposed a distributed architecture with workers, a durable queue, and mechanisms for recovering jobs after failures. The interviewer challenged how I’d prevent duplicate execution, and we spent a considerable amount of time discussing leases and idempotent processing.

I thought the conversation had gone reasonably well. Then he asked how many scheduled tasks their actual product processed. I said I didn’t know. He told me the current workload was around 40,000 tasks per day.

I looked at the architecture I’d drawn and realized it was far more complicated than the problem required. A database-backed scheduler with careful concurrency control might have handled the workload perfectly well, depending on latency requirements and operational constraints. I’d designed a system for an imaginary company instead of the one interviewing me.

The recruiter sent a rejection two days later. I closed my laptop and wondered whether I’d become one of those engineers who knew how to discuss architecture but had forgotten how to solve ordinary problems.

The Eighth Company Had a Very Different Problem

The job description was unusually direct.

The company operated a business platform that had been developed over more than a decade. Its backend contained older Java applications alongside newer Spring Boot services, and the engineering team needed someone who could improve reliability without interrupting customers. They weren’t promising a complete rewrite or a glamorous cloud migration.

They wanted an engineer who could make existing software safer to change.

I applied because I’d spent much of my career doing exactly that.

The recruiter warned me that the technical interview would involve reviewing legacy code. She said some candidates found the exercise frustrating because there wasn’t a clean architecture or an obvious correct answer. I told her that sounded considerably more familiar than designing another social media platform from scratch.

The interviewer was a Principal Engineer named Thomas. He’d joined the company six years earlier and had been responsible for several attempts to modernize the application. After a brief introduction, he opened the enormous Java class and asked me to review a method responsible for updating customer account balances.

The Code Was Ugly. The Business Rules Were Worse.

The method accepted an account ID, a transaction amount, and an operation type. It queried the database, calculated the new balance, inserted an audit record, and updated several related tables. Everything happened inside a single method with multiple conditional branches.

I noticed several problems immediately.

The SQL was difficult to read, the exception handling was inconsistent, and there were repeated queries that could probably be consolidated. A few methods had misleading names, and one helper returned null when it couldn't find an account.

Thomas asked which issue I’d fix first.

I was tempted to suggest splitting the class into smaller services and introducing a proper repository layer. That would’ve made the code easier to understand, but I didn’t yet know whether the application had reliable tests or what other systems depended on its behavior. So I asked how the account balances were reconciled.

Thomas looked interested.

He explained that a nightly process compared account records with transaction history and reported discrepancies. The reconciliation job had been introduced years earlier after a production incident, and the team still relied on it to identify certain classes of errors. I asked whether the application had ever produced an incorrect balance during concurrent updates.

He said yes.

Then he showed me the incident report.

A Bug That Had Survived Three Refactoring Attempts

Two requests could update the same account within milliseconds of each other. Each request read the current balance, calculated a new value, and wrote the result back to the database. Under certain timing conditions, one update could overwrite the effect of another.

The team had attempted to fix the problem several times.

One developer had introduced synchronization around the Java method. That reduced the issue during local testing but didn’t protect against concurrent requests handled by different application instances. Another engineer had proposed moving the operation into a separate service, which wouldn’t have addressed the database concurrency problem by itself.

The third attempt had introduced a retry mechanism.

Unfortunately, the retry logic repeated the entire operation without a stable identifier for the original transaction. Under some failure conditions, that created the possibility of applying the same business operation more than once.

Thomas asked what I’d recommend.

I told him I wanted to understand the database constraints and transaction boundaries before changing the application code. We discussed row-level locking, conditional updates, and optimistic concurrency control. I explained that whichever approach we chose would need to preserve the business invariant and behave correctly when multiple application instances processed requests simultaneously.

He asked whether I’d use synchronized.

I said it wouldn’t be sufficient for the deployment architecture he’d described.

He asked whether I’d introduce Kafka. I said I couldn’t see a reason to add it yet. Thomas smiled. Apparently, several previous candidates had proposed replacing the entire balance update mechanism with an event-driven architecture.

The Question That Changed the Interview

After about forty minutes, Thomas asked how I’d deploy a fix to the application.

I explained that I’d first want a reproducible test demonstrating the concurrency problem. I’d also examine the existing reconciliation process and identify whether there were historical inconsistencies that needed separate correction. Once the fix was implemented, I’d test it against concurrent requests and verify that failed operations didn’t produce duplicate or partially applied updates.

He asked whether I’d deploy the change on a Friday afternoon.

I said that depended on the urgency of the incident and the team’s ability to monitor and recover from the release. If customers were actively losing money because of incorrect balances, delaying the fix could be worse than deploying outside the normal release window. Otherwise, I’d prefer a controlled rollout with enough engineers available to investigate unexpected behavior.

Thomas asked what I’d do if the fix required changing a database constraint.

I said I’d want to inspect existing data before introducing the constraint. A migration could fail if historical records violated the new rule, and that wasn’t something I’d want to discover halfway through a production deployment. We’d need to understand the migration’s locking behavior and have a recovery plan appropriate to the database version and table size.

We spent another twenty minutes discussing the rollout.

Nobody asked me to explain the difference between an abstract class and an interface. Nobody asked me to design a system that handled a billion requests per second. We were discussing how to change a dangerous piece of software without making customers pay for our mistakes.

It was the most enjoyable interview I’d had in months.

Thomas Told Me Why They’d Stopped Trying to Rewrite Everything

Near the end, I asked whether the company planned to replace the application entirely.

He said they’d tried twice.

The first attempt involved extracting several business workflows into microservices. After nine months, the team had discovered that much of the application behavior depended on undocumented interactions between modules. The new services were cleaner, but reproducing the existing business rules required far more work than anyone had estimated.

The second attempt focused on replacing the database access layer.

That project made progress, but the team struggled to demonstrate that the new implementation preserved the behavior customers depended on. They eventually reduced the scope and concentrated on the parts of the application causing the most incidents.

Thomas said he’d become skeptical of engineers who promised a complete rewrite before understanding the system.

I understood why.

I’d worked on projects where replacing old code looked straightforward until someone discovered a business rule that hadn’t been documented in years. The code might have been ugly, but it represented decisions made under constraints that were no longer visible. Sometimes those decisions were wrong. Sometimes they were the only reason the system still worked.

The Offer Arrived Six Days Later

The recruiter called the following week and said the technical feedback had been strong. Thomas had appreciated my approach to the concurrency problem and the fact that I’d spent time understanding the existing system before proposing changes. The hiring manager wanted one final conversation about team responsibilities and compensation.

That discussion was straightforward.

The company needed someone comfortable working across older Java applications and newer Spring Boot services. They also wanted help improving testing, reviewing database changes, and reducing recurring production incidents. I asked about the team’s release process, on-call responsibilities, and how much freedom engineers had to improve existing code.

The offer arrived six days after my technical interview. It wasn’t the highest salary I’d discussed during my job search, but it was competitive, and the responsibilities were clear. More importantly, the team seemed to value the kind of engineering work I’d spent years learning to do.

I accepted.

What Seven Rejections Had Made Me Forget

I’d spent two months preparing as though senior engineering interviews were competitions to demonstrate the most advanced architecture. I studied distributed systems, cloud infrastructure, and concurrency because those subjects matter, but I’d gradually started assuming that simple solutions made me look inexperienced.

That assumption was wrong.

The eighth company had a real concurrency problem in a business-critical application. The difficult part wasn’t choosing an impressive technology; it was understanding the failure conditions, preserving the business rules, and deploying a fix without introducing new inconsistencies. Those decisions required technical depth, even if the final implementation used familiar database mechanisms.

I’d also forgotten how much of senior engineering involves understanding software that already exists.

Most companies have systems that carry years of assumptions, integrations, and compromises. Engineers who can investigate those systems carefully and improve them without unnecessary disruption provide real value. The work doesn’t always produce an exciting architecture diagram, but customers notice when their transactions stop failing.

I still study modern backend technologies.

I still care about distributed systems and system design. But I no longer assume that the most complicated solution is the one an experienced engineer should propose. The eighth company didn’t need me to invent a new architecture. They needed someone willing to open a 1,400-line Java class, understand why it existed, and make one dangerous operation reliable. After seven rejections, that was the interview I finally knew how to pass.

Prepare for Senior Java and Backend Engineering Interviews

If you’re preparing for Senior Java or Spring Boot interviews, my Senior Java & Spring Boot Engineering OS focuses on the problems experienced backend engineers actually face, including transactions, concurrency, database behavior, and production reliability. You can also start with my Free Senior Engineer Interview & Production Playbook if you want practical technical scenarios before committing to a more extensive preparation resource.

For engineers preparing for architecture interviews, the Senior & Staff System Design Interview OS covers distributed systems, reliability, and the trade-offs behind production-ready designs. I write more about engineering careers, interview experiences, and production systems on Substack, where I share the kinds of lessons that rarely appear in traditional interview preparation.

I’m also building ProdRescue AI System Design Lab, an interview practice platform focused on realistic system design scenarios and feedback for experienced engineers. If you’d like to help shape the product or get considered for early access, you can join the private beta research list here.

I Failed 7 Senior Java Interviews. Then One Company Asked Me to Fix a 12-Year-Old Application was originally published in Engineering Under Pressure on Medium, where people are continuing the conversation by highlighting and responding to this story.

I forgot a basic Java concept, argued with an interviewer, and walked away convinced I’d ruined my chances. Five days later, the recruiter called with an offer.

I knew I’d failed the interview when I started explaining Java’s volatile keyword and couldn't remember whether it made a compound operation atomic. The interviewer had asked a reasonable question about a shared counter, and I was halfway through an answer that sounded increasingly confident and increasingly wrong. I had seven years of backend experience, but apparently that wasn't enough to stop me from embarrassing myself over something I'd learned early in my career.

The interviewer, a Staff Engineer named Ryan, waited until I’d finished. Then he asked whether count++ was safe if the variable was declared volatile. I said yes, paused, and immediately realized what I'd done. Visibility wasn't the same thing as atomicity, and I'd just confused the two in front of someone evaluating me for a Senior Backend Engineer position.

I corrected myself, but the damage felt permanent. We still had forty minutes remaining, and I spent most of them wondering whether Ryan had already decided to reject me. By the time the call ended, I’d mentally added another failure to a spreadsheet that was becoming unpleasant to open.

It was my tenth interview in forty-five days.

I’d received seven rejections, withdrawn from one process, and was waiting for feedback from another. The position with Ryan’s company was one of the few I genuinely wanted, which made the mistake considerably harder to forget.

My First Job Search in Years Was Going Badly

I’d left my previous role after spending almost five years working on Java and Spring Boot services. Much of my experience involved transaction processing, PostgreSQL performance, and production incidents that rarely resembled the clean problems found in interview preparation guides. I knew how to investigate a slow service, trace a failed payment, and recognize when a proposed architecture would become expensive to maintain.

What I hadn’t done in years was sit across from another engineer and answer technical questions under a timer.

My first interview involved implementing a rate limiter. I understood the token bucket algorithm but struggled when the interviewer introduced concurrent requests and asked me to modify the implementation. The second company rejected me after a system design round, even though I’d spent most of the hour explaining how to scale a notification service.

The third interview was particularly frustrating. The hiring manager asked about a migration I’d led, and I described the technical decisions in detail. His feedback said I hadn’t demonstrated enough leadership, which surprised me because I’d been responsible for coordinating the migration across several services.

By the fifth interview, I was preparing almost every evening. I’d review Java concurrency, PostgreSQL transaction isolation, Redis caching, and distributed systems before moving on to coding exercises. I was solving more problems than I had during the previous two years combined, but the results weren’t improving.

The rejections also started affecting how I behaved during interviews.

I became impatient when someone asked a question I considered too basic. If an interviewer challenged my answer, I’d spend too much time defending it. I was so worried about appearing inexperienced that I occasionally made simple questions unnecessarily complicated.

Interview Eight Was the Lowest Point

The eighth company was hiring a Senior Backend Engineer for a payment infrastructure team. The interviewer asked me to design a workflow that could recover when an external provider processed a payment but failed to return a response. I’d encountered similar problems in production, so I was comfortable discussing durable state, idempotency, and reconciliation.

The technical conversation went well until he asked how I’d handle an engineer who disagreed with my design.

I described an architecture review where I’d convinced another developer to adopt an asynchronous processing approach. The interviewer asked whether the other engineer had eventually agreed with me. I said we’d reached a consensus, although that wasn’t entirely accurate.

He asked what had happened after deployment.

The system had experienced several operational problems, including duplicate message processing and difficult recovery scenarios. Some were directly related to concerns the other engineer had raised during the review. I explained those problems, but I could tell the interviewer was becoming more interested in how I’d handled the disagreement than in the architecture itself.

He asked whether I’d admitted the other engineer had been right. I said we’d discussed the lessons afterward. He asked whether I’d actually told him he was right. I hadn’t. The rejection arrived three days later. The recruiter said the team wanted someone who demonstrated stronger collaboration during technical disagreements. I didn’t enjoy reading that feedback, but it was difficult to argue with.

I Almost Canceled Interview Ten

The ninth company had put its hiring process on hold after my technical round. I’d passed the initial screening, but the recruiter couldn’t tell me when the team would resume interviews. That left me with seven rejections, one withdrawal, and one process that might never restart.

The tenth interview was scheduled for the following Tuesday.

I considered canceling because I was exhausted and didn’t think another interview would change much. The company was building software for financial operations, and the role involved maintaining Java services that processed customer transactions. It was exactly the kind of work I understood, but I’d already been rejected by companies with similar requirements.

I decided to attend because I’d already taken the afternoon off.

Ryan joined the call a few minutes late and apologized. He explained that he’d been helping another engineer investigate an incident involving database connections. We talked briefly about the issue, and I mentioned a similar problem I’d encountered when a Spring service held transactions open during external API calls. He seemed interested. Then he opened the coding exercise.

The Java Question I Should Have Answered Correctly

Ryan showed me a small class containing a shared counter. Multiple threads could increment the counter, and the requirement was to return an accurate total after all operations completed. The code used a volatile int, and he asked whether the implementation was thread-safe.

I said it was.

He asked why.

I started explaining how volatile ensured visibility across threads, which was correct. Then I claimed that it also made the increment operation safe, which wasn't. I knew that incrementing a counter involved reading the current value, calculating the next value, and writing it back.

Those steps weren’t atomic. Ryan asked whether two threads could read the same value before either wrote the result. I looked at the code again and realized my mistake. I told him I’d confused visibility with atomicity and that volatile alone wouldn't prevent lost updates.

He asked what I’d use instead.

I suggested AtomicInteger for the simple counter, then discussed when synchronization might be necessary if multiple related operations needed to happen together. He asked whether I'd use an atomic counter for a financial account balance, and I explained why the answer depended on where the authoritative state lived and what consistency guarantees were required. The conversation recovered, but I was still embarrassed. I apologized for the initial answer. Ryan shrugged and said he’d rather hear someone correct a mistake than spend ten minutes defending it. That should have made me feel better. It didn’t.

Then We Disagreed About a Database Transaction

The second exercise involved a Spring Boot service that transferred money between two accounts. Ryan showed me a method that loaded both accounts, adjusted their balances, and saved the updated entities. The method was annotated with @Transactional.

He asked whether the implementation was safe under concurrent requests.

I said no.

Two transfers could read the same account balance and calculate updates based on stale values. Depending on the database behavior and how the entities were updated, the system could experience lost updates or other consistency problems. Ryan asked how I’d fix it. I suggested introducing appropriate locking or conditional updates, depending on the expected workload and transaction requirements. He asked whether I’d use optimistic locking, and I said it could work if conflicts were detected and retried safely.

Then he proposed using a distributed Redis lock around the transfer operation.

I disagreed.

I explained that introducing Redis as the primary mechanism protecting account balances would create another failure dependency. If the lock expired while a transfer was still running, another request might enter the critical section. I’d rather enforce the essential consistency rules in the database responsible for storing the balances. Ryan argued that database locking could reduce throughput.

I agreed, but I didn’t think throughput was the first concern without knowing the workload. We needed to understand the transaction volume, contention patterns, and acceptable failure behavior before choosing a concurrency strategy. We spent nearly fifteen minutes discussing it. I expected him to become irritated.

Instead, he kept asking questions.

Eventually, he said he wasn’t necessarily recommending Redis. He wanted to see whether I’d challenge an architecture decision when I believed it introduced unnecessary risk. I laughed because I’d spent the previous fifteen minutes assuming we were having an argument.

I Left the Call Convinced I’d Failed

The interview ended after seventy minutes. Ryan thanked me for the discussion and said he’d send his feedback to the hiring team. He didn’t tell me whether I’d passed, and I didn’t ask.

I closed my laptop and immediately wrote down the volatile mistake.

It bothered me more than the database discussion because it was such a basic concept. I’d worked with Java concurrency for years, yet I’d answered too quickly and confused two different guarantees. I started wondering how many other fundamentals I’d become careless about. That evening, I reviewed the Java Memory Model and wrote a small example demonstrating lost updates with a volatile counter. I also revisited the database transaction exercise and tested a simplified version using PostgreSQL. I wanted to understand the details rather than simply remember which answer an interviewer preferred.

Two days passed without feedback. On Friday morning, the recruiter emailed asking whether I had time for a short call. I assumed she wanted to reject me politely.

The Recruiter Had Different News

She said the technical feedback had been positive and the team wanted to move forward with an offer. I asked whether there would be another interview, and she explained that Ryan had recommended skipping the remaining technical round because the discussion had covered the areas they needed to evaluate. I was surprised enough that I asked what he’d said. She told me he appreciated how I’d handled the concurrency problem after recognizing my mistake. He’d also liked the database discussion because I’d challenged an assumption and explained the operational consequences without becoming defensive.

I told her I’d thought the interview had gone badly.

She laughed and said candidates often misjudged their own performance.

The offer arrived the following Monday. It included a base salary slightly above my target and responsibilities that matched the production engineering work I wanted to do. I asked a few questions about on-call expectations and the team’s approach to incident reviews before accepting. I still thought about the Java mistake. But I was beginning to understand why Ryan hadn’t treated it as the defining moment of the interview.

The Difference Between Being Wrong and Being Difficult

During my earlier interviews, I’d spent too much energy trying to avoid mistakes. I prepared polished explanations, memorized common questions, and became defensive whenever someone challenged an answer. I wanted interviewers to see an experienced engineer who always knew what he was doing.

That wasn’t how I behaved during real production incidents.

When a service failed at 2 AM, I didn’t have the luxury of defending an incorrect assumption because it sounded impressive. I had to inspect the evidence, reconsider my understanding, and work with other engineers until we identified the problem. Sometimes a junior developer noticed something I’d missed, and the correct response was to listen.

The tenth interview was the first time I’d behaved that way under pressure. I made a basic mistake, acknowledged it, and corrected my reasoning. Later, I disagreed with a Staff Engineer about a technical decision and explained why I thought the proposed approach was risky. Neither conversation was particularly polished, but both reflected how I’d approach the work.

That didn’t mean technical correctness was optional.

If I’d continued insisting that volatile made increments atomic, the interview probably would've ended differently. The distinction was that I recognized the error and could reason through the correct behavior once it became apparent. The database discussion mattered for the same reason. I wasn’t trying to prove that Ryan was wrong. I was explaining why I’d be cautious about using a distributed lock to protect financial state and what evidence I’d want before making that decision.

What I’d Do Differently Before My Next Interview

I’d still review Java fundamentals, database transactions, and system design problems. Those skills matter, and the mistakes I’d made during earlier interviews showed that preparation was necessary. But I’d spend less time memorizing ideal answers and more time practicing how to reason when an interviewer introduced something unexpected.

I’d also become more comfortable saying that I needed to reconsider an answer.

There is a difference between not knowing a concept and making an incorrect assumption under pressure. Interviewers need to evaluate technical competence, but they also need to understand how someone behaves when their first idea doesn’t work. An engineer who can identify a mistake and adjust their approach is easier to collaborate with than someone who treats every correction as a personal attack.

I’d stop assuming that a difficult conversation meant the interview was going badly.

Some of the most useful technical discussions I’ve had involved disagreement. The important part was whether the conversation helped us understand the problem and make a better decision. If I could explain my reasoning clearly and remain open to evidence, I didn’t need everyone to agree with me immediately. The tenth interview wasn’t my cleanest performance.

I forgot a basic Java rule, spent fifteen minutes disagreeing with a Staff Engineer, and left convinced I’d embarrassed myself. Five days later, I had an offer from the company I most wanted to join. I wish I’d understood earlier that a senior engineer’s value isn’t measured by how confidently he answers every question. It’s also visible in what he does after discovering that his answer was wrong.

Prepare for Senior Backend and Software Engineer Interviews

If you’re preparing for Senior Java or Backend Engineer interviews, my Free Senior Engineer Interview & Production Playbook covers practical technical scenarios and production engineering concepts. For deeper Java preparation, the Senior Java & Spring Boot Engineering OS focuses on backend fundamentals, concurrency, transactions, and real production problems.

For advanced architecture rounds, the Senior & Staff System Design Interview OS covers distributed systems, reliability, and technical trade-offs. I also publish engineering career stories and technical breakdowns on Substack, where I write about the decisions and mistakes that shape experienced engineers.

I Did 10 Software Engineer Interviews in 45 Days. The Best Offer Came After My Worst Interview. was originally published in Engineering Under Pressure on Medium, where people are continuing the conversation by highlighting and responding to this story.

I spent 52 days preparing for coding interviews, system design rounds, and behavioral questions. Nobody had asked me to explain a decision I’d made three years earlier. I wasn’t ready for how much that conversation would bother me.

The interviewer opened my GitHub repository about twenty minutes into our call. I’d included the project on my résumé because it was one of the few public examples of my backend work, although I hadn’t touched it in almost three years. It was a small order management application built with Java, Spring Boot, and PostgreSQL, originally written during a period when I was trying to learn more about transaction processing.

He scrolled past the README, opened the commit history, and selected a change from March 2023. The commit message was fix order processing issue, which told him approximately nothing about what I'd fixed. He clicked through to a service method where I'd removed a database transaction annotation and changed the way orders were updated.

“Do you remember why you made this change?” he asked. I told him I vaguely remembered an issue involving transaction boundaries, but I’d need a minute to look at the surrounding code. He said that was fine and shared the screen so we could review it together.

I was expecting another interview where someone wanted me to design a system for ten million users. Instead, a stranger was asking me to explain a decision made by a younger version of myself, back when I thought every difficult production problem could be solved by adding another annotation.

The Previous Eight Interviews Had Worn Me Down

I’d started looking for a new position after leaving a backend engineering job that had become increasingly repetitive. Most of my recent work involved maintaining existing services, investigating payment processing failures, and helping other engineers understand code that had survived several architectural migrations. I wanted a role with more technical ownership, preferably at a company where senior engineers still spent a meaningful amount of time working directly with software.

The first two interviews went badly for familiar reasons. One company wanted much deeper Kubernetes experience than the job description suggested, while another asked me to implement a graph algorithm I hadn’t practiced since university. I didn’t particularly enjoy either interview, but I understood why I hadn’t advanced.

The third rejection bothered me more. I’d spent almost an hour designing a background processing system, discussing retry policies, worker failures, and how to prevent duplicate business operations. The interviewer seemed engaged throughout, yet the recruiter later said my architecture hadn’t demonstrated enough experience with large-scale systems.

By interview five, I was spending nearly every evening studying. My browser contained articles about PostgreSQL isolation levels, Java concurrency, distributed locks, and the kinds of system design questions that appear in expensive preparation courses. I’d started recording myself answering behavioral questions because I was convinced that sounding more confident would improve my results.

It didn’t.

The seventh company rejected me after a hiring manager conversation, and the eighth ended when the position was put on hold. I was beginning to wonder whether I should lower my salary expectations or apply for mid-level positions. My previous experience still felt valuable to me, but eight unsuccessful processes had made it surprisingly difficult to talk about that experience without sounding defensive.

The Ninth Interview Started With an Ordinary Conversation

The company built software for logistics businesses, including services responsible for shipment tracking, invoicing, and customer notifications. Their backend stack was close to mine, and the role involved improving existing systems rather than building an entirely new platform. That appealed to me because I’d spent enough time around production software to know that keeping something reliable can be more difficult than creating the first version.

My interviewer was a Staff Engineer named Andrew. He’d joined the company when the engineering team was small and had apparently spent much of the previous year untangling a service that several teams depended on. We talked briefly about my experience with Spring Boot and the kinds of incidents I’d investigated before he asked whether we could look at something I’d written.

I assumed he meant a coding exercise. Instead, he opened the public repository listed on my résumé and asked whether I was comfortable discussing it. I told him the code was old and that I’d probably approach some parts differently today, which was true but also a convenient way of lowering expectations.

Andrew said he wasn’t interested in whether the project used the latest framework version. He wanted to understand how I thought about software when I was responsible for maintaining it. Then he opened the commit that had removed @Transactional from an order processing method.

The Commit I Couldn’t Explain

The original implementation had created an order, reserved inventory, and updated the payment state inside a single service method. I’d placed @Transactional on the method because I wanted the database operations to succeed or fail together. At some point, I'd removed the annotation while trying to resolve an issue involving an external inventory provider.

Looking at the code now, I could see why the design was dangerous. The method called an external HTTP service while a database transaction was active, potentially keeping a connection occupied while waiting for a network response. If that dependency slowed down, the application could exhaust its connection pool even when PostgreSQL itself was performing normally.

I explained this to Andrew, and he agreed that the transaction boundary deserved attention. Then he asked whether removing the annotation had actually fixed the consistency problem. I looked at the updated method and realized that it hadn’t, because several database writes could now commit independently.

An order could be created successfully while a later inventory update failed. The method caught certain exceptions and returned an error response, but that response didn’t undo changes that had already been committed. I’d replaced one operational risk with another and apparently considered the issue resolved.

Andrew asked whether I’d written a regression test.

I searched the repository and found a test added in the same commit. It checked that the method returned an error when the inventory provider failed, but it didn’t verify the database state afterward. The test passed because it asserted the response, while the inconsistency remained invisible.

I remember feeling embarrassed in a way that was different from failing a coding problem. This wasn’t an unfamiliar algorithm or a tricky interview puzzle. It was my own code, and I’d left a defect in it because the test had confirmed what I wanted to believe.

He Didn’t Ask Me to Defend the Decision

Andrew gave me a few minutes to inspect the surrounding implementation. I expected him to ask for an ideal architecture involving queues, durable events, and distributed transactions. I’d practiced that conversation enough times that I could probably have delivered a convincing explanation without looking at the code.

Instead, he asked what I’d do if this repository belonged to a service currently running in production. Customers were placing orders, the inventory provider occasionally timed out, and the engineering team had limited time for changes. He wanted to know what I’d investigate before touching the implementation.

I told him I’d start by understanding whether the inconsistent states had actually occurred and how the business recovered from them. I’d look at order records, inventory reservations, error logs, and the behavior of any scheduled reconciliation jobs. I also wanted to know whether the external provider supported idempotent reservation requests, because retrying a timed-out operation without understanding its outcome could make the problem worse.

He asked what I’d deploy first.

I said I’d be reluctant to redesign the entire workflow before understanding the existing recovery behavior. If the evidence showed that transactions were holding database connections during external calls, I’d want to address that risk, but moving the call outside the transaction would require a durable way to track incomplete operations. Depending on the provider’s guarantees, that might involve an explicit reservation state and a recovery process that could safely resume after a crash.

Andrew listened and asked several questions about failure scenarios. We discussed what would happen if the application crashed after the provider reserved inventory but before the local database recorded the result. I explained that a local transaction couldn’t make the external operation atomic and that we’d need a way to identify uncertain outcomes rather than assuming every timeout meant failure.

For the first time in weeks, I wasn’t trying to remember the answer to an interview question. I was trying to understand a problem.

The Conversation Became Uncomfortable Again

About forty minutes into the interview, Andrew returned to the original commit. He asked whether I thought the younger engineer who’d written it had understood the consequences of removing the transaction. I wanted to say that I’d been working quickly and that the project hadn’t been particularly important, but neither explanation answered his question.

I admitted that I probably hadn’t understood the failure modes well enough. I’d recognized that holding a transaction open during a network call was risky, but I’d treated removing the annotation as a complete fix. The test I’d written had reinforced that assumption because it checked the visible error response without checking whether the underlying data remained consistent.

Andrew asked whether I’d made similar mistakes in professional projects.

I told him about a payment processing incident where our team had initially treated provider timeouts as failed transactions. The implementation had worked during ordinary testing, but an intermittent network problem exposed the fact that some payments could succeed externally without being recorded correctly. We’d eventually introduced better operation tracking and reconciliation, although the first version had been more complicated than it needed to be.

He wanted to know what I’d personally contributed to the fix. I explained the parts I’d implemented and which decisions had come from other engineers, including a colleague who’d identified a failure scenario I’d overlooked. I didn’t try to make the story more impressive than it was, partly because I’d already spent forty minutes being reminded how unconvincing that could become.

Andrew told me he’d seen experienced engineers make similar mistakes. What concerned him more was when someone couldn’t explain how their understanding had changed. He said the repository was useful precisely because it showed a decision I might no longer agree with.

I Left Without Knowing Whether I’d Passed

The interview ran eleven minutes over schedule. Andrew apologized for keeping me longer than planned and said the recruiter would contact me after he’d submitted his feedback. We hadn’t discussed algorithm complexity, drawn a distributed architecture, or completed a conventional coding exercise.

I closed the meeting and sat at my desk for a while.

Part of me thought I’d done terribly. I’d shown an interviewer a real bug in my own code, admitted that my test was inadequate, and spent several minutes explaining why a decision I’d made three years earlier was wrong. Compared with the polished answers I’d prepared for my previous interviews, it felt like an unusually poor performance.

But another part of me felt relieved.

I’d spent nearly two months trying to demonstrate that I knew everything expected of a senior engineer. During this interview, I’d been allowed to say that I didn’t remember something, investigate it, and change my assessment when the evidence contradicted my assumptions. That felt much closer to the work I’d actually done throughout my career. The recruiter called the following afternoon and invited me to the final round.

The Final Round Wasn’t About GitHub

The Engineering Manager asked about technical ownership and how I’d approach improving a service that nobody on the team wanted to touch. I described a project where we’d inherited a difficult codebase with limited tests and several undocumented dependencies. Instead of proposing a rewrite, we’d started by identifying the most frequent incidents and adding enough observability to understand where the service was failing.

She asked how I’d convinced the team to invest time in work that didn’t immediately deliver new features. I explained that we’d connected the reliability problems to customer impact and the engineering time spent investigating recurring incidents. The improvements had been incremental, and some were difficult to justify until we could show how often the same failures were interrupting other work.

We talked about the difference between making a system easier to understand and making it architecturally impressive. I told her I’d become more skeptical of large rewrites because I’d seen how much behavior was hidden inside old services. She seemed interested in that answer and asked how I’d decide when a rewrite really was justified.

The company made an offer four days later.

It wasn’t an extraordinary compensation package, and the position didn’t come with a fashionable Staff Engineer title. The work involved a complicated backend platform, a team that seemed comfortable discussing mistakes, and enough ownership to make the role interesting. I accepted after reviewing the responsibilities and speaking with two engineers who would be my colleagues.

What I Changed About Interview Preparation

I still think coding exercises and system design practice are useful. Engineers need to reason about algorithms, database behavior, concurrency, and the consequences of architectural decisions. I wouldn’t recommend abandoning that preparation just because one unusual interview went well.

But I had spent almost no time reviewing my own work.

I’d practiced explaining systems designed for millions of users while struggling to remember why I’d changed a transaction boundary in a project I’d written myself. I’d memorized answers about technical leadership without revisiting the decisions that had actually shaped my judgment. The contrast was uncomfortable, especially because my Git history contained more evidence of my engineering development than any polished interview story.

Before my next interview process, I’d review a few projects I’d genuinely worked on and identify the decisions I’d reconsider today. I’d look at old pull requests, incident reports, and migrations where the original plan changed after implementation began. Those examples would give me something concrete to discuss when an interviewer asked about mistakes, trade-offs, or how I’d handled uncertainty.

I’d also be more careful about what I put on my résumé. If I describe myself as someone who improved transaction reliability or led a database migration, I should be able to explain what I changed and why. The interviewer might never ask to see the code, but I ought to understand the decisions well enough to defend them without relying on a prepared speech.

The ninth interview didn’t erase the frustration of the previous eight. Some of those companies had evaluated skills I genuinely needed to improve, and others simply weren’t a good fit. What changed was my understanding of what I could bring into an interview beyond a collection of correct answers. Three years earlier, I’d made a mistake in a small Spring Boot project and committed it with a useless message. I never imagined that someone would eventually open that commit during a job interview. I’m glad he did, although I wish I’d written a better test.

Prepare for Your Next Senior Engineer Interview

If you’re preparing for Senior Backend Engineer interviews, my Free Senior Engineer Interview & Production Playbookcovers practical technical scenarios and production engineering problems. For more demanding architecture rounds, the Senior & Staff System Design Interview OS helps you practice explaining design decisions, failure scenarios, and trade-offs that don’t have a single correct answer.

I also write about software engineering careers, difficult technical decisions, and lessons from production systems on Substack. If you’re preparing for interviews while trying to become a better engineer, those are the conversations I want to keep having.

I Got Rejected From 8 Software Engineer Interviews. The Ninth Asked About My Git History was originally published in Engineering Under Pressure on Medium, where people are continuing the conversation by highlighting and responding to this story.

The automation saved $420,000 a year on paper. Nobody had calculated what an autonomous mistake might cost.

At 9:06 on a Thursday morning, our CEO walked into the engineering meeting carrying a spreadsheet that had apparently changed his understanding of human civilization. Customer support represented one of the company’s largest operational expenses, and an AI vendor had promised to automate most of the repetitive work. The presentation included impressive response-time charts, a projected annual saving of $420,000, and enough references to autonomous agents to make ordinary software sound embarrassingly unemployed.

Six weeks later, twelve support positions had been eliminated through a restructuring. The company introduced an AI agent that answered billing questions, processed refund requests, updated customer accounts, and escalated conversations it could not resolve. Leadership celebrated the deployment as evidence that the business could scale without increasing headcount.

For the first month, the dashboard looked excellent. Average response time fell from several hours to less than a minute, ticket closure rates increased, and the number of cases assigned to human operators dropped sharply. Our CEO described the project as the most successful operational transformation in the company’s history.

Then a customer named Rebecca requested a complete record of changes made to her enterprise subscription. Her account had been downgraded, two authorized users had lost access, and a billing credit had been applied without her approval. The AI agent had performed all three actions during a conversation about an invoice discrepancy.

Rebecca wasn’t particularly interested in our automation strategy. She wanted to know who had authorized the changes, what information the system had used, and whether the company could reconstruct the decisions. Her email included the phrase formal dispute, which caused the legal department to discover an unexpected enthusiasm for engineering documentation.

Our Staff Engineer, Marcus, was asked to investigate. He requested the agent’s execution trace, tool invocation history, authorization context, and original customer messages. The support platform could show a conversation summary and the final account state, but it could not reconstruct the complete sequence of decisions.

The CEO asked whether Marcus could fix the logging problem before the customer escalated. Marcus explained that the problem extended beyond logging because the system had been allowed to perform consequential actions without sufficiently constrained authority.

The AI had been given permission to solve customer problems. Nobody had defined precisely which problems it was allowed to solve by changing customer data.

AI Agent Automation Creates a Different Kind of Production Risk

Traditional customer support automation often follows predefined workflows. A refund request might trigger a rules engine that checks eligibility, calculates an amount, and asks for confirmation before applying the transaction. The workflow can be complex, but the important decisions are represented explicitly in application logic.

An AI agent introduces a different execution model. It interprets language, selects tools, evaluates responses, and may decide which action to take next. That flexibility can improve the customer experience, especially when requests are difficult to categorize.

It also creates additional failure modes. The agent may misunderstand a request, rely on incomplete context, select an inappropriate tool, or take an action that is technically permitted but inconsistent with the customer’s actual intent. The risk becomes more serious when tool access includes account modifications, refunds, or permission changes.

Our implementation exposed several internal APIs through an agent orchestration layer. The agent could retrieve invoices, modify subscription plans, issue limited credits, and update customer information. Each API had been developed for use by trusted internal applications, and the integration team had focused primarily on making the tools accessible to the model.

Marcus discovered that authorization checks were based largely on the service account used by the AI platform. The underlying APIs could verify that the integration was authenticated, but they did not consistently enforce whether the specific customer had authorized the requested operation.

The distinction was critical. An authenticated agent is still capable of making an unauthorized decision. Identity and permission checks must apply to the actual operation, including the customer, resource, and action being requested.

The company had built a capable automation system. It had also created a new path through which misunderstood language could become an irreversible business operation.

The Customer Never Asked to Downgrade Her Subscription

Marcus reconstructed the incident using the records that were available. Rebecca had contacted support because her latest invoice contained a charge she did not recognize. She asked the agent to explain the amount and determine whether it was correct.

The agent retrieved her subscription details and identified an optional feature associated with the charge. It then inferred that removing the feature would resolve the billing concern. During the conversation, the agent offered to help reduce future charges, and Rebecca responded that she wanted the invoice corrected.

The model interpreted that response as permission to change the subscription. It called the plan modification API, removed the optional feature, and applied a credit. Because the feature supported access for additional users, the change also affected account permissions.

The system had completed a sequence of valid API calls. The failure was in the decision that connected the customer’s words to those calls. Rebecca had asked for an explanation and correction of a disputed charge, while the agent had treated the conversation as authorization for a subscription downgrade.

Marcus found that the tool descriptions encouraged the agent to resolve billing issues proactively. The prompts emphasized customer satisfaction and minimizing escalation, but the workflow did not require explicit confirmation before modifying a paid subscription.

The agent had followed the incentives built into its instructions. It resolved the ticket quickly, avoided human escalation, and recorded the interaction as successful. The customer discovered the consequences only after colleagues reported losing access.

The support dashboard counted the case as an automated resolution. Rebecca counted it as a company changing her contract without permission.

Both metrics appeared in the same quarterly report, although only one had been shown to investors.

AI Agent Guardrails Must Exist Outside the Prompt

The engineering team initially proposed improving the system prompt. They wanted to add instructions telling the agent never to modify a subscription without explicit customer consent. Marcus agreed that clearer instructions could reduce mistakes, but he rejected the idea that prompt wording alone was an adequate authorization control.

Language models can misunderstand ambiguous requests, and their behavior can vary with context. Even a carefully written prompt cannot provide the same deterministic guarantees as an authorization system. The application must enforce important restrictions regardless of what the model decides.

Marcus proposed separating conversational reasoning from business action authorization. The agent could analyze a request and propose a subscription change, but the backend would require an explicit confirmation record before executing the operation. The confirmation would need to identify the account, requested change, and relevant consequences.

The design also introduced risk-based approval requirements. Low-impact informational requests could be handled automatically, while account changes with significant financial or access consequences required stronger controls. Certain operations would require human review.

The goal was to preserve the useful flexibility of the AI agent while preventing it from becoming the final authority over sensitive actions. The model could recommend an operation, but the application would determine whether the operation was permitted.

Marcus summarized the principle during the architecture review. The agent was allowed to reason about customer intent, but authorization needed to remain an enforceable property of the system.

The CEO asked whether that would make the automation slower. Marcus said some operations would require additional confirmation, which was preferable to discovering unauthorized changes through customer complaints.

Audit Logs Are Part of the Product, Not an Optional Debugging Feature

Rebecca’s request exposed another weakness. The platform stored conversation summaries and some application events, but it did not maintain a complete audit trail connecting the original request to the actions performed. The company could identify the final account changes but struggled to explain the decision sequence.

Marcus proposed a structured execution record for every consequential agent operation. Each record would include the authenticated customer identity, agent execution identifier, proposed action, authorization result, tool invocation, and resulting state change. Sensitive data would be handled according to the company’s retention and access policies.

The team also needed to distinguish model-generated reasoning from verifiable execution evidence. Internal model reasoning is not a dependable audit artifact, and some model providers do not expose it. Instead, the system should record observable inputs, selected tool calls, policy evaluations, approvals, and outcomes.

A simplified audit event might look like this:

{
  "executionId": "agent-run-82f4",
  "customerId": "customer-1042",
  "action": "subscription.change",
  "resourceId": "subscription-8871",
  "requestedChange": {
    "removeFeature": "additional-seats"
  },
  "authorization": {
    "status": "DENIED",
    "reason": "EXPLICIT_CONFIRMATION_REQUIRED"
  },
  "toolExecuted": false
}

The example showed a proposed action rejected by a deterministic policy check. In a production system, the audit event would also need reliable persistence, appropriate access controls, and correlation with relevant application events. The exact schema would depend on the organization’s compliance and operational requirements.

Marcus recommended that audit records be protected against unauthorized modification. A system responsible for explaining sensitive account changes should not allow ordinary application components to rewrite the evidence after the fact.

The team also reviewed data retention policies. Keeping complete conversations indefinitely could create privacy and security risks, while retaining too little information could undermine incident investigation. The design needed to balance traceability with data minimization and applicable obligations.

The company had previously considered audit logging an internal engineering concern. Rebecca had demonstrated that it was also a customer trust concern.

Tool Permissions Were More Dangerous Than the Model’s Answers

The investigation revealed that the agent had broad access to internal support APIs. The integration used a service identity with permission to modify subscriptions, issue credits, and update customer records. The original design assumed that the agent would use those permissions appropriately because its prompts described the intended workflows.

Marcus considered that assumption unsafe. A tool-using agent should receive the minimum permissions necessary for the current operation. The application should validate the requested action independently, including the identity and authority of the customer involved.

The team introduced narrower tool interfaces. Instead of exposing a general subscription modification API, the agent received tools with constrained operations and explicit input validation. The backend also enforced limits on financial adjustments and required confirmation for changes affecting account access.

For example, a refund tool could accept a transaction identifier and proposed amount, but the backend would verify refund eligibility and permitted limits. The agent could not bypass those checks by constructing a persuasive explanation. The same principle applied to subscription changes and user permissions.

Marcus also reviewed the possibility of prompt injection. Customer messages, retrieved documents, and external content could contain instructions attempting to redirect the agent’s behavior. The system needed to treat those sources as untrusted data rather than allowing them to override application policies.

This was especially important because the agent retrieved information from multiple internal systems. A malicious or simply malformed document could influence the model’s interpretation of a request. Strong tool authorization reduced the consequences of such manipulation.

The team began treating the AI agent as an untrusted decision-making component operating within a trusted application boundary. That design was less glamorous than the original promise of fully autonomous support, but it provided a clearer foundation for production reliability.

AI Automation ROI Looked Different After the Incident

The CEO had justified the project using projected labor savings. The original financial model assumed that automated ticket resolution would reduce support costs without materially increasing engineering or operational expenses. The early dashboard appeared to support that assumption.

Marcus asked finance to revisit the calculation. The system now required additional engineering work for authorization controls, audit logging, monitoring, incident investigation, and human escalation. Those costs were part of operating the automation safely.

The team also needed to measure incorrect resolutions. A ticket closed automatically was not necessarily a ticket resolved correctly. Some customers might reopen cases, contact another support channel, or abandon the service after receiving an unsatisfactory answer.

The original dashboard focused on automation rate and average response time. Marcus recommended adding measures for confirmed resolution quality, customer corrections, unauthorized action attempts, and the rate of cases requiring human intervention after an automated response.

Finance discovered that the projected $420,000 annual saving was a gross estimate based on reduced staffing costs. It did not account for all implementation, oversight, and failure-related expenses. The actual net benefit would depend on the reliability of the automation and the cost of maintaining appropriate controls.

The CEO was unhappy with the revised model. He argued that the company had already demonstrated substantial efficiency gains. Marcus agreed that AI had reduced the effort required for many routine support tasks, but he wanted the financial evaluation to reflect the full operating system.

Automation can produce genuine savings. Those savings become less convincing when the organization ignores the cost of correcting mistakes or maintaining the controls needed to prevent them.

The spreadsheet remained useful. It simply needed more rows than the CEO had originally anticipated.

Human-in-the-Loop AI Doesn’t Mean Humans Review Everything

The incident prompted a debate about whether every AI agent action should require human approval. Some executives favored strict review because they wanted to avoid another customer dispute. Others worried that extensive approvals would eliminate the productivity benefits of automation.

Marcus proposed a risk-based approach. Informational responses could be automated when supported by reliable data and appropriate safeguards. Actions with limited, reversible consequences could proceed within defined policy boundaries. High-impact changes required explicit customer confirmation or human review.

The design considered reversibility, financial impact, customer permissions, and uncertainty. A request to explain an invoice was different from a request to cancel a subscription. A small billing credit within an approved policy was different from modifying an enterprise contract.

Human review was reserved for situations where the potential consequences justified the additional cost. The application would also escalate cases when required information was missing or when the customer’s intent remained ambiguous. The goal was to make escalation a controlled part of the workflow rather than a sign that automation had failed.

Marcus emphasized that human reviewers needed useful context. Sending an employee a vague request to approve an agent decision would create another rubber-stamping process. Review interfaces should present the proposed action, supporting evidence, policy checks, and relevant customer confirmation.

The company also needed to measure reviewer workload and decision quality. Human oversight could introduce delays and errors if poorly designed. The system should support informed decisions rather than assuming that inserting a person automatically guarantees correctness.

The resulting architecture was less autonomous than the original marketing presentation. It was also better aligned with the company’s actual risk tolerance.

Production Monitoring for AI Agents Needs Different Metrics

Traditional application monitoring remained important. The team continued tracking API latency, error rates, dependency health, and infrastructure utilization. However, those metrics could not reveal whether an AI agent had misunderstood a customer or selected an inappropriate action.

Marcus introduced monitoring for agent-specific behavior. The system recorded tool invocation rates, policy denials, escalation frequency, execution failures, and changes in the distribution of actions. These signals helped engineers identify unexpected behavior that might not appear as a conventional application error.

The team also developed evaluation datasets representing realistic support scenarios. Cases included ambiguous billing questions, subscription modifications, permission changes, and requests containing conflicting information. The evaluations checked whether the agent proposed appropriate actions and respected authorization boundaries.

Offline evaluations were useful before deployment, but production monitoring remained necessary. Customer language and business conditions could differ from the test dataset, and model or prompt changes could alter behavior. The team introduced staged rollouts and regression testing for changes to agent configuration.

Marcus also required versioning for prompts, tool schemas, and policy definitions. When an incident occurred, engineers needed to know which configuration had been active. Without that information, reproducing agent behavior could become difficult.

The company had previously monitored whether the agent completed its tasks. It now monitored whether those tasks were appropriate, authorized, and consistent with business rules.

That was a more demanding definition of success, but it reflected what customers actually expected.

Staff Engineers Must Understand the Business Consequences of AI

Marcus’s role extended beyond fixing a technical bug. He needed to explain why the original architecture created unacceptable risk and help leadership redesign the operating model. The challenge involved engineering, product policy, customer trust, and financial expectations.

He avoided arguing that AI agents were inherently unsuitable for customer support. The technology had demonstrated value in answering routine questions and reducing response times. The failure emerged from granting broad authority without sufficient controls.

Marcus proposed a narrower initial scope for autonomous actions. The agent would continue handling informational requests and low-risk workflows while sensitive operations received stronger authorization and audit requirements. The organization could expand automation as it gathered evidence that the controls were effective.

The CEO eventually accepted the revised approach. He continued describing the initiative as an AI transformation, although the engineering team privately referred to it as the expensive discovery of access control. The company restored the affected customer’s subscription and reviewed similar agent actions for potential problems.

The incident also changed how new automation projects were evaluated. Teams needed to identify which decisions the model would make, what tools it could invoke, and which operations required deterministic safeguards. Financial projections had to include ongoing oversight and operational costs.

Marcus’s contribution was valuable because he connected the technology to the consequences of using it. He understood that a successful API call could still represent a failed business decision.

That distinction is becoming increasingly important as AI systems move from generating text to taking actions.

AI Agents Are Software Systems With Authority Problems

The promise of AI agents is compelling. They can interpret natural language, coordinate workflows, and automate tasks that previously required human judgment. In suitable environments, they can improve responsiveness and reduce repetitive work.

However, autonomous action introduces responsibilities that ordinary chat interfaces do not carry. An agent that modifies customer accounts or financial records must operate within enforceable permissions. Its actions must be observable, auditable, and recoverable where possible.

Prompt instructions can guide behavior, but they should not replace authorization policies. Tool access should be constrained, and sensitive operations should require appropriate confirmation. Monitoring must evaluate the correctness and consequences of actions rather than simply measuring completion rates.

Organizations also need realistic economic models. Automation may reduce certain labor costs while increasing requirements for engineering, evaluation, and oversight. The net benefit depends on whether the system delivers reliable outcomes at an acceptable total cost.

Marcus’s company eventually built a safer automation platform. It continued using AI for customer support, but the system no longer treated every plausible interpretation of a request as permission to change customer data.

Rebecca received the audit information the company could substantiate, along with an explanation of the correction. The missing historical details could not be reconstructed perfectly, which became part of the incident record and the reason stronger logging was introduced.

The CEO still liked the original spreadsheet. Marcus preferred the new authorization service because it was considerably less likely to generate an emergency meeting with legal.

Frequently Asked Questions About AI Agents in Production

What are the biggest risks of autonomous AI agents?

Autonomous AI agents can misunderstand instructions, select inappropriate tools, or perform actions without sufficient authorization. Risks become more significant when agents modify financial records, customer accounts, or other sensitive data.

How do you secure AI agent tool calls?

Tool calls should pass through deterministic authorization and validation controls. The application should verify the authenticated user, permitted operation, resource scope, and required approvals independently of the model’s instructions.

What are AI agent guardrails?

AI agent guardrails are controls that constrain model behavior and the actions an agent can perform. They can include prompt guidance, tool restrictions, policy enforcement, validation, monitoring, and human approval workflows.

Why are audit logs important for AI agents?

Audit logs help organizations reconstruct what actions an agent proposed and executed. They support incident investigation, accountability, customer disputes, and compliance obligations where applicable.

Can prompt engineering prevent unauthorized AI actions?

Prompt engineering can improve behavior but cannot provide deterministic authorization guarantees. Sensitive actions require application-level controls that enforce permissions regardless of model output.

What is human-in-the-loop AI automation?

Human-in-the-loop automation involves human review or approval at selected points in an automated workflow. The appropriate level of oversight depends on the risk, impact, and reversibility of the action.

How do companies calculate AI automation ROI?

AI automation ROI should consider implementation costs, ongoing infrastructure, engineering maintenance, human oversight, error correction, and measurable business benefits. Gross labor savings alone may not represent the net financial outcome.

How do you monitor AI agents in production?

Production monitoring should include application reliability metrics and agent-specific signals such as tool calls, policy denials, escalation rates, action outcomes, and evaluation results. Configuration versioning and execution traces also support investigation.

What is prompt injection in AI agents?

Prompt injection occurs when untrusted content attempts to influence a model’s instructions or behavior. Agents that consume external documents or user messages need safeguards to prevent that content from overriding trusted policies.

What should Staff Engineers know about AI agent architecture?

Staff Engineers should understand tool authorization, workflow orchestration, auditability, failure handling, evaluation, observability, and risk-based approval. They should also be able to connect technical design choices to business consequences.

Free: The Engineer’s High-Stakes Playbook

AI agents are creating new opportunities for automation, but the underlying engineering challenges remain familiar. Authorization, data consistency, failure handling, and observability determine whether automated workflows can operate safely in production.

I created a free engineering playbook covering production investigation, system design reasoning, and critical career decisions. It’s a practical starting point for engineers who want to strengthen their ability to reason about complex software systems.

Get the Free Engineer’s High-Stakes Playbook

Build the Engineering Skills Behind Reliable AI Systems

If you’re preparing for Senior or Staff Engineer roles, understanding production architecture is increasingly valuable. AI systems still depend on backend services, databases, authorization policies, and distributed workflows that require careful design.

The Senior Backend Production Engineering Operating System: 30 Playbooks | Java, Spring Boot & Debugging is a resource focused on Java, Spring Boot, debugging, and operational reliability. It is designed for engineers who want structured preparation around complex production systems.

For broader technical interview preparation, the Senior & Staff Engineer Interview Operating System: 15 Playbooks | System Design, Coding & Leadership provides a resource covering system design, coding, and leadership expectations. It can help experienced engineers organize preparation for advanced technical roles.

Help Shape an AI-Powered System Design Interview Tool

I’m researching an interactive AI-powered system design interview platform for engineers preparing for Senior, Staff, and Principal roles. The proposed platform would allow candidates to practice architecture decisions, respond to realistic interviewer follow-ups, and receive structured feedback on scalability, reliability, data consistency, and technical tradeoffs.

The product is still in the research stage, and feedback from practicing engineers will help determine which capabilities matter most. If you’d like to contribute your perspective or express interest in early testing, you can complete the short survey.

Take the System Design Interview Research Survey

I also write about AI systems, software engineering careers, backend architecture, production reliability, and the decisions facing experienced engineers. My Substack is where I continue those discussions and publish more detailed perspectives on engineering work.

Follow My Engineering Writing on Substack

Our CEO Replaced 12 Support Agents With AI. Then a Customer Asked for the Audit Log was originally published in Javarevisited on Medium, where people are continuing the conversation by highlighting and responding to this story.

I Got Rejected After 5 Software Engineer Interviews. Then I Saw the Salary They Offered My Replacement

I spent six weeks interviewing for a Senior Backend Engineer role. The company said my salary expectations were too high. Three months later, I learned what they were willing to pay.

The recruiter called me on a Friday afternoon, eleven days after my final interview. We’d spent six weeks going through technical assessments, system design discussions, and conversations with engineering leadership. I was sitting at my kitchen table with my laptop open, already thinking about how I’d negotiate the offer. She sounded uncomfortable.

The hiring team had decided to move forward with another candidate. She said they’d appreciated my technical experience, particularly my background in Java, PostgreSQL, and distributed systems. Unfortunately, my compensation expectations were higher than the company could accommodate.

I asked whether there was room to negotiate.

She said the budget was fixed.

I’d requested $175,000 in total annual compensation. The recruiter had told me during our first conversation that the company was targeting somewhere between $150,000 and $180,000, depending on experience. I’d passed every technical round and had seven years of backend engineering experience, so I didn’t think my expectation was unreasonable. I thanked her and ended the call.

Three months later, I found out they’d hired someone for approximately $205,000.

I wasn’t supposed to know that.

The Interview Process Had Started Perfectly

The company was building infrastructure for subscription billing and payment processing. Its engineering team had grown quickly, and several services were struggling with the operational consequences of that growth. The position involved improving reliability, redesigning parts of the transaction processing workflow, and helping less experienced engineers make better architectural decisions.

That sounded like work I’d already been doing.

The first interview was with a recruiter named Emily. She asked about my previous roles, the systems I’d worked on, and the kinds of problems I wanted to solve next. We discussed compensation early because I didn’t want to spend weeks interviewing for a position that couldn’t meet my expectations. I told her I was targeting $175,000. She said that was within the range they’d discussed internally. I remember that detail because I wrote it down immediately after the call. The technical interviews started the following week.

The First Technical Round Was About a Production Incident

The interviewer showed me an architecture involving a Spring Boot service, PostgreSQL, and an external payment provider. Customers occasionally received duplicate charges when requests timed out, and the engineering team wanted to understand how I’d investigate the problem.

I started by asking how payment attempts were identified and whether the provider supported idempotency keys. We discussed what happened when the provider successfully processed a payment but the application failed before recording the result. I explained why blindly retrying an uncertain operation could create duplicate business effects.

The interviewer then asked how I’d redesign the workflow.

I proposed durable payment state, stable operation identifiers, and a reconciliation process for uncertain outcomes. We also discussed database transactions and why external provider calls shouldn’t be treated as though they participated in the same atomic transaction.

The interview lasted fifty-five minutes. At the end, he told me he’d enjoyed the discussion and appreciated that I hadn’t immediately proposed replacing the entire architecture. The recruiter emailed the next morning to schedule the second round.

The System Design Interview Was Harder

The next interviewer was a Staff Engineer who’d been at the company for almost four years. He asked me to design a system that could process subscription renewals for millions of customers while handling payment failures, retries, and billing adjustments.

I’d practiced similar problems, but he kept introducing constraints that made the design more complicated.

At one point, he asked what would happen if a customer changed subscription plans while a renewal payment was being processed. I explained how I’d represent the billing state and separate the calculation of an invoice from the attempt to collect payment. We discussed how to avoid charging customers based on inconsistent subscription information.

He challenged my decision to introduce asynchronous processing for part of the workflow.

I explained that the billing calculation could happen independently from payment collection, but that the system still needed a clear source of truth for the invoice and its state. We talked about what should happen when a worker crashed after a provider had accepted a payment. I didn’t answer everything perfectly. There were several moments when I had to reconsider my assumptions, and one proposed approach introduced more complexity than the workload justified. The interviewer pointed that out, and I simplified the design.

The following day, Emily told me the feedback had been positive.

She scheduled a conversation with the Engineering Manager.

The Manager Asked Why I Wanted to Leave

The Engineering Manager, David, seemed more interested in my previous team than in my technical knowledge. He asked about the projects I’d owned, how I handled disagreements, and what I’d do if another senior engineer strongly opposed an architecture decision.

I described a migration that had gone badly.

We’d introduced asynchronous communication between services before fully understanding the operational requirements. The migration eventually succeeded, but it took longer than expected and created additional failure scenarios. I explained what I’d underestimated and how the team had adjusted the plan. David asked whether I’d make the same decision again. I said probably not. He seemed satisfied with that answer.

Near the end, he asked what would make me accept an offer from the company.

I talked about the engineering problems, the opportunity to influence architecture, and the importance of working with people who could challenge my decisions. Compensation mattered too, but I’d already discussed my expectations with Emily. David nodded and said the team was excited about my background.

I left the call thinking the process was almost finished.

Then the Salary Conversation Changed

Two days later, Emily called and asked whether my compensation expectations were flexible.

I reminded her that we’d discussed $175,000 during our first conversation.

She said the company was reviewing its budget and wanted to understand whether I’d consider something closer to $150,000. I asked what had changed. She said the hiring team was evaluating candidates at different experience levels.

That surprised me because the position had been advertised as Senior Backend Engineer from the beginning. The responsibilities hadn’t changed, and nobody had suggested that my technical performance was below the expected level.

I told her I’d consider the complete offer, including benefits and equity, but that $150,000 was below the range I was targeting. She said she’d discuss it with the team. I didn’t hear anything for eight days. When she finally called, the company had decided to hire someone else.

I Thought I’d Made a Negotiation Mistake

For the next few weeks, I kept wondering whether I should have handled the conversation differently. Maybe I’d been too direct about compensation, or perhaps the company had found someone with stronger technical experience. I knew hiring decisions involved more than interview performance, but the timing made it difficult not to connect the rejection to salary.

I replayed the conversation with Emily several times.

I’d been polite and flexible about the structure of the offer. I hadn’t demanded an immediate decision or threatened to withdraw. I’d simply said that $150,000 was below my target after the recruiter had originally confirmed that $175,000 was within range.

Still, I wondered whether I’d lost a good opportunity over $25,000.

Then I started interviewing with other companies.

One offered $165,000, but the role involved more on-call responsibilities than I wanted. Another offered $180,000 with a smaller equity component. I eventually accepted a position with a team whose technical expectations and compensation were clearer from the beginning. I stopped thinking about the original company. Until I received a message from someone who worked there.

Three Months Later, I Learned What Happened

A former colleague had joined the company shortly after my interview process ended. We hadn’t spoken in several months, and he contacted me after seeing that I’d started a new role. We talked about the engineering team, the projects he was working on, and how the company was handling its recent growth. Then he mentioned the new Senior Backend Engineer. I asked whether that was the position I’d interviewed for.

He said yes.

The engineer had joined from a larger technology company and had extensive experience with payment infrastructure. My colleague mentioned that the hire had been expensive, somewhere around $205,000 in annual compensation.

I asked whether that included equity. He said he wasn’t certain about the exact breakdown. I didn’t press him for more details.

The number was secondhand, and I had no way to verify the compensation package. It was possible the company had changed its budget, adjusted the role, or offered a different combination of salary and equity. But it still bothered me. The company had told me it couldn’t accommodate $175,000. A few months later, it appeared willing to spend considerably more.

The Part That Took Me Longer to Understand

My first reaction was anger.

I’d spent six weeks interviewing, taken time off for technical rounds, and prepared extensively for a role that supposedly couldn’t meet my compensation expectations. If the company had wanted someone with a different background, I would’ve preferred to hear that directly. But I also knew I didn’t have the full picture.

The new engineer might have brought experience the company valued more highly. The hiring team might have reconsidered its requirements after interviewing several candidates. The compensation figure might have included equity or other components that made the comparison misleading. What bothered me wasn’t necessarily that another engineer had been paid more.

It was that the company had framed the decision as a fixed budget constraint after previously telling me my expectations were within range. That made the rejection harder to interpret. I’d treated the salary discussion as a negotiation between two parties who understood the same numbers. The company might have been using compensation as one factor in a broader hiring decision without explaining the rest.

I couldn’t know for certain.

What I’d Do Differently in My Next Salary Negotiation

I’d ask for the compensation range in writing before beginning a lengthy interview process. A verbal range is useful, but it doesn’t establish how the company evaluates candidates within that range. I’d also ask whether the budget depends on leveling, because a Senior Engineer title can mean very different things across organizations.

I’d avoid treating a recruiter’s initial confirmation as a guarantee.

Hiring teams sometimes change their assessment of a candidate’s level during interviews, and compensation decisions can involve approvals outside the recruiter’s control. If the range changes midway through the process, I’d ask what specifically changed and whether the responsibilities or leveling had been adjusted.

I’d also be careful about negotiating against myself.

When Emily asked whether I’d accept $150,000, I could have immediately lowered my expectations out of fear of losing the opportunity. Instead, I explained my target and said I’d evaluate a complete offer. I still think that was a reasonable response. The company was entitled to choose another candidate. I was entitled to decide what compensation made the role worthwhile. The difficult part was accepting that a rejection doesn’t always provide enough information to understand what happened.

The Offer I Eventually Accepted

My new role involved backend infrastructure, PostgreSQL performance, and production reliability. The team was smaller than the one I’d originally interviewed with, but the engineering problems were interesting and the responsibilities were clearly defined.

The hiring process took three weeks.

The recruiter shared the compensation range before the first technical interview, and the final offer fell within it. I negotiated a modest increase in base salary, reviewed the on-call expectations, and accepted after speaking with the engineers I’d be working alongside.

I don’t know whether the first company would’ve been a better place to work.

Maybe the new engineer was exactly what they needed. Maybe their budget changed after my interviews. Maybe I’d misunderstood how they evaluated candidates. What I do know is that I spent weeks questioning my value because one company told me my expectations were too high. Three months later, I learned that the number itself might never have been the real problem. I wish I’d understood that sooner.

Prepare for Senior Software Engineer Interviews

If you’re preparing for Senior Backend or Staff Engineer positions, my Free Senior Engineer Interview & Production Playbook covers practical interview preparation and production engineering scenarios. For advanced technical rounds, the Senior & Staff System Design Interview OS focuses on architecture decisions, distributed systems, and the trade-offs experienced engineers are expected to explain.

I also publish engineering career stories, interview lessons, and technical breakdowns on Substack. If you’re navigating difficult hiring processes or preparing for your next senior engineering role, you’ll find more practical discussions there.

I Got Rejected After 5 Software Engineer Interviews. Then I Saw the Salary They Offered My Replaceme was originally published in Engineering Under Pressure on Medium, where people are continuing the conversation by highlighting and responding to this story.

ChatGPT helped me solve problems faster, understand system design, and prepare answers that sounded impressive. I didn’t realize how dependent I’d become until an interviewer asked me to explain 14 lines of Java.

The interviewer shared a Java method and asked me what was wrong with it. There were fourteen lines of code, a HashMap, and a loop that counted how many times each customer ID appeared in a list. I'd written more complicated code before breakfast at my previous job.

I looked at the method for about thirty seconds and told him it had a potential concurrency problem. He asked why I thought multiple threads would be accessing it. I said they might, depending on how the method was used, and started explaining how I’d replace the map with a concurrent collection. He waited until I finished. Then he pointed at the loop and asked what happened when the input contained a null customer ID. I stared at the screen.

The method unboxed a nullable Integer into an int. A null value would throw a NullPointerException before the map operation even happened. It was a simple bug, the kind I'd probably have caught during an ordinary code review.

But I’d been so eager to demonstrate senior-level knowledge that I’d missed it. I failed the interview. That evening, I opened ChatGPT to ask it what I’d done wrong. Halfway through typing the prompt, I realized how ridiculous that was.

Six Weeks Earlier, AI Had Made Me Feel Unstoppable

I’d started using AI heavily after leaving my previous backend engineering job. I had several years of Java and Spring Boot experience, but the interview market felt different from the one I’d entered earlier in my career. Companies were asking about distributed systems, cloud infrastructure, coding exercises, and technical leadership in the same hiring process.

I wanted to prepare efficiently.

Every morning, I gave ChatGPT a topic and asked it to generate senior-level interview questions. I’d work through PostgreSQL isolation levels, Kafka consumer failures, Spring transaction behavior, and common system design problems. When I got stuck, I’d ask for an explanation and compare it with my own understanding.

At first, it was genuinely useful. I could review unfamiliar concepts much faster than before, and the explanations often helped me connect ideas I’d encountered in production. I started asking AI to critique my answers, generate follow-up questions, and show me how a Staff Engineer might explain the same architecture.

Then I made a small change to my routine. Instead of attempting a problem before asking for help, I started requesting the ideal solution first. I told myself I was learning from better examples. In reality, I was getting very good at recognizing answers I’d already seen.

The First Four Interviews Didn’t Expose the Problem

My first interview was for a Senior Backend Engineer position at a financial technology company. The system design round involved a payment processing service, and I’d prepared almost the exact problem with AI the previous night. I discussed idempotency keys, durable transaction state, and reconciliation after uncertain provider responses.

The interviewer seemed satisfied.

I advanced to the next round, where a hiring manager asked about a production incident I’d handled. That conversation went well too, although the company eventually rejected me because they wanted someone with more experience leading cross-team projects.

The second interview involved a URL shortener. I’d practiced that architecture several times, so I explained caching, database partitioning, and redirect performance without much hesitation. I felt confident until the interviewer asked whether the company actually needed the infrastructure I’d proposed. I adjusted the design, but I was noticeably slower once the conversation moved away from the version I’d prepared. The third company asked me to implement a rate limiter. I recognized the problem immediately and wrote a token bucket implementation. The interviewer introduced a concurrency requirement, and I struggled to modify the code without changing several assumptions I’d made at the beginning.

The fourth interview ended after a coding round involving a small scheduling service. I could explain the solution afterward. Writing it from scratch was another matter. I’d been studying the right topics, but something about my preparation wasn’t translating into independent problem-solving.

I Started Recognizing a Pattern in My Own Behavior

One Saturday, I sat down to implement a simple LRU cache without using AI. I’d solved the problem before and understood why a hash map combined with a doubly linked list could provide constant-time access and eviction. I opened my editor.

Twenty minutes later, I was still adjusting the linked-list operations.

I kept making mistakes when moving nodes, updating pointers, and handling the empty-list case. None of those problems was particularly difficult, but I was frustrated by how slowly I was working. I opened ChatGPT and pasted the implementation. Within seconds, it identified the pointer update I’d missed. I fixed the code and felt relieved. Then I realized I’d done the same thing three times that morning.

Every time I encountered friction, I handed the problem to AI before spending enough time understanding it myself. The tool was helping me finish exercises, but I wasn’t developing the patience required to debug my own work. That distinction hadn’t mattered much while I was studying. It mattered considerably during interviews.

Interview Five Was the First Serious Warning

The fifth company asked me to review a Spring Boot service that processed customer refunds. The interviewer shared a method annotated with @Transactional that updated a refund record and called an external payment provider.

I recognized the transaction boundary problem immediately.

I’d discussed similar examples with AI dozens of times. I explained that holding a database transaction open during an external network call could increase connection usage and create difficult failure scenarios. I also mentioned idempotency and reconciliation.

The interviewer asked what would happen if the provider completed the refund but the application crashed before committing the database transaction. I gave the correct high-level answer.

The system could end up with an external refund that wasn’t reflected in its local state. Then he asked me to modify the code.

I hesitated.

I knew the architecture should separate durable local state from external processing, but I hadn’t practiced implementing the workflow. I’d spent most of my preparation reading explanations and reviewing completed examples. The interviewer gave me fifteen minutes to make a reasonable improvement. I wrote something that partially addressed the problem, but it still had an inconsistent failure path. He pointed it out, and I tried to repair the implementation while explaining the design.

The feedback arrived three days later. My conceptual knowledge was strong, but my implementation skills hadn’t met their expectations. I was annoyed because I’d understood the problem. Looking back, understanding it wasn’t enough.

Interview Six Was the One That Hurt

The sixth company was hiring a Senior Java Engineer for a backend platform team. The technical round started with a code review exercise involving collections, null handling, and a few deliberately misleading implementation details.

I expected something much harder.

The interviewer showed me the fourteen-line method that counted customer IDs. I immediately started looking for advanced problems because I’d spent weeks preparing for concurrency, distributed transactions, and complex system design questions.

I suggested replacing HashMap with ConcurrentHashMap.

He asked what evidence suggested concurrent access.

I started talking about shared state. He asked whether the map was shared. It wasn’t. The map was created inside the method and returned after the loop completed. The actual problem was a nullable value being unboxed without validation. I eventually found it, but only after the interviewer pointed me toward the relevant line.

He asked me to write a test demonstrating the failure. I could do that. Then he asked why I’d spent several minutes discussing concurrency when the code didn’t contain shared mutable state. I didn’t know how to answer without admitting I’d been trying to sound more sophisticated than the problem required. The interview ended after forty minutes. The rejection arrived the next afternoon.

I Didn’t Stop Using AI. I Changed When I Used It.

I considered abandoning AI tools for interview preparation entirely. That would’ve been an easy reaction, but it wouldn’t have addressed the actual problem. AI hadn’t prevented me from thinking; I’d developed a habit of asking it to think before I’d made a serious attempt myself. So I changed my routine. For coding problems, I worked alone for a fixed period before requesting hints. If I got stuck, I wrote down what I’d tried and what I believed was failing. Only then would I ask AI to challenge my reasoning or identify an assumption I’d missed.

I also stopped asking for complete solutions unless I’d already finished the exercise.

When reviewing Java code, I started with ordinary correctness questions before considering advanced architectural concerns. I checked null handling, mutation, exceptions, collection behavior, and whether the implementation actually satisfied the stated requirements.

For system design preparation, I practiced explaining my own architecture before comparing it with AI-generated alternatives. The difference was uncomfortable. Some problems took twice as long. I made more mistakes. But I could finally tell which parts I understood and which parts I’d only recognized from previous explanations.

The Seventh Interview Felt Slower

The seventh company asked me to implement a small in-memory reservation system. It needed to track available inventory, reserve items, and reject requests when there wasn’t enough stock.

I started by clarifying the expected behavior.

The interviewer asked me to assume a single process initially, with concurrency discussed afterward. I implemented the basic operations, wrote tests for insufficient inventory, and checked that failed reservations didn’t modify the stored state.

I made a mistake while updating the inventory count. Normally, I’d have copied the code into an AI tool and asked it to identify the bug. Instead, I walked through the state changes with a concrete example and found that I was updating the available quantity before validating the full request.

I corrected it.

The interviewer asked how I’d handle concurrent reservations.

I explained that the current implementation wasn’t safe under concurrent updates and discussed how synchronization or an atomic database operation could protect the inventory invariant. I didn’t pretend the in-memory version was ready for distributed deployment. The interview took almost the full hour. I advanced. It was the first time in weeks that I’d finished a technical round feeling confident about what I actually understood.

Interviews Eight Through Eleven Were Uneven

The eighth company rejected me after a system design round. I struggled to explain how I’d migrate a large PostgreSQL table without creating unacceptable downtime. I knew the general approach, but my experience with that specific migration pattern was limited.

The ninth company asked me to debug a Java service experiencing high memory usage. I investigated object retention, cache growth, and heap behavior, but I missed an important clue in the provided metrics. The interviewer eventually showed me where my reasoning had gone wrong.

I didn’t pass that round either.

Those rejections were disappointing, but they felt different.

I could identify the technical gaps without convincing myself that the interviewers had misunderstood my experience. I went back to studying the areas where I’d struggled and practiced explaining them without relying on prepared answers.

The tenth interview involved designing a background job processing system. I clarified the execution guarantees, discussed retries and duplicate processing, and proposed a design that matched the team’s workload. I advanced to the final round, but the company paused hiring before making a decision. The eleventh interview was for a Senior Backend Engineer position at a SaaS company. The interviewer asked me to review a service that processed subscription renewals.

I recognized several problems, but I didn’t start listing technologies.

I asked about the expected behavior when a payment provider returned a timeout and whether repeated renewal requests could occur. We discussed how the system represented uncertain outcomes and what guarantees the external provider offered. Then I wrote a small implementation. It wasn’t perfect, but I could explain every decision. The interviewer challenged one of my assumptions, and I changed the code without needing to reconstruct the entire solution.

I advanced to the next round.

The Twelfth Interview Was the First Time I Felt Like Myself Again

The final company asked me to investigate a production incident involving a Spring Boot service and PostgreSQL. Request latency had increased sharply, but CPU usage remained normal. The interviewer gave me access to connection pool metrics, application traces, and a simplified version of the affected code.

I worked through the evidence.

The service was holding database transactions open while waiting for an external dependency. Connection acquisition time had increased, causing requests to queue even though the application wasn’t consuming much CPU.

I explained how I’d confirm the diagnosis and what I’d consider before changing the transaction boundaries. We discussed the consistency problems that could arise when external operations were separated from local database updates.

The interviewer asked how I’d test the fix. I described integration tests for failure scenarios and load testing to verify that connection usage remained stable under the expected workload. I also explained which metrics I’d monitor after deployment. The company made an offer the following week. I accepted after reviewing the role and compensation.

What Those Twelve Interviews Taught Me About AI

I still use AI when writing code.

It’s useful for exploring unfamiliar APIs, reviewing implementation options, and challenging assumptions. It can help explain complex technical concepts and identify mistakes that would otherwise take longer to investigate.

But I’ve become much more careful about using it during learning.

When I’m preparing for an interview, I need to know whether I can solve the problem without assistance. Reading a correct explanation isn’t the same as producing one, and recognizing a familiar architecture doesn’t mean I can adapt it when the requirements change.

The most dangerous part of my earlier preparation was how productive it felt.

I completed more exercises, reviewed more topics, and generated better-looking answers than ever before. My notes were impressive, and I could discuss advanced engineering concepts with confidence. Yet I was becoming less comfortable with the slow, frustrating process of figuring something out myself. That was what the fourteen-line Java method exposed. I hadn’t forgotten how to program. I’d become too accustomed to skipping the part where I had to struggle.

The twelfth interview didn’t make me a fundamentally different engineer. It showed me that I could still investigate problems, write code, and explain decisions when I gave myself enough practice doing those things independently. I continue using AI almost every day. I just make sure I’m still the one doing the engineering.

Prepare for Senior Software Engineer Interviews

If you’re preparing for Senior Backend or Software Engineer interviews, my Free Senior Engineer Interview & Production Playbook covers practical technical scenarios and production engineering concepts. For engineers working with Java and Spring Boot, the Senior Java & Spring Boot Engineering OS focuses on backend fundamentals, production problems, and the technical decisions experienced engineers need to understand.

For advanced architecture preparation, the Senior & Staff System Design Interview OS covers distributed systems, reliability, and system design trade-offs. I also publish engineering career stories and technical breakdowns on Substack, where I write about the parts of software engineering that rarely appear in interview preparation guides.

I Used AI to Prepare for 12 Software Engineer Interviews. Then I Failed the Easiest One. was originally published in Engineering Under Pressure on Medium, where people are continuing the conversation by highlighting and responding to this story.

By my ninth interview, I was tired of explaining my experience to people who seemed to have already made up their minds. Then an interviewer asked me about the worst six months of my career.

I nearly canceled the interview on Thursday morning. It was scheduled for 3 PM, and I had already spent the first half of the day answering a recruiter who wanted another copy of my résumé, even though I’d uploaded it twice. The company was smaller than the others I’d interviewed with, the salary range was lower than I’d hoped, and I was tired of pretending I was excited about every opportunity.

I’d been unemployed for forty-three days. In that time, I’d completed eight software engineering interviews, received six rejections, and watched two companies disappear after promising to follow up. My savings weren’t disappearing at a terrifying rate yet, but I’d started checking my bank balance more often than I wanted to admit.

The interview was for a Senior Backend Engineer position. The job description mentioned Java, PostgreSQL, AWS, and experience maintaining production systems. I’d done that work for years, although lately it felt like every recruiter wanted someone who had invented distributed computing personally.

I opened the meeting link at 2:59 PM.

The interviewer joined three minutes late, apologized because he’d been dealing with an incident, and asked whether I minded if he finished eating his lunch during the call. His name was Michael, and he looked like someone who’d spent the morning arguing with a database.

I liked him immediately.

The First Eight Interviews Had Made Me Difficult to Interview

My first rejection had come from a company where I’d passed the coding assessment but failed the system design round. They asked me to design a notification service, and I spent forty minutes discussing queues, retries, and horizontal scaling. The feedback said I hadn’t demonstrated enough ownership, which was interesting because nobody had asked me about ownership.

The second company rejected me after a behavioral interview. The third wanted someone with more Kubernetes experience, despite advertising the position as a Java backend role. By the fourth interview, I’d started recognizing the phrases recruiters used when they wanted to reject someone without explaining why.

I also became increasingly defensive.

When an interviewer questioned an architecture decision, I assumed he was trying to catch me making a mistake. When someone asked why I’d chosen PostgreSQL instead of another database, I gave a five-minute explanation of transactions and consistency. I answered questions as though every conversation were an argument I needed to win.

The worst interview was number six.

The interviewer asked me about a project that hadn’t gone well. I described a migration that had taken longer than expected because another team kept changing its requirements. That explanation was technically accurate, but it left out the part where I’d ignored two warnings from one of our senior engineers.

He asked whether I’d contributed to the delay. I said we’d all made mistakes. He asked what mine had been. I couldn’t give him a straight answer. The rejection arrived the next morning.

Michael Didn’t Ask Me to Introduce Myself

He started by asking what kind of work I’d been doing before my job search. I told him about a payment processing platform I’d helped maintain, including the parts responsible for transaction state, asynchronous processing, and reconciliation. He seemed interested when I mentioned that the system had been through several migrations. Then he asked what had been the hardest period. I expected him to ask about a production outage. Instead, he said he wanted to know about a period when I’d felt like I wasn’t doing particularly good work.

That was a different question.

I thought about giving him the usual answer about a difficult deadline or a demanding project. I’d prepared several versions of those stories, each ending with a successful deployment and a valuable lesson. They sounded convincing when I practiced them alone. But Michael was still eating his sandwich, waiting for me to answer, and I was too tired to perform another carefully edited version of my career.

So I told him about the six months when I’d been a terrible engineer to work with.

The Project That Made Me Miserable

It happened during my fourth year as a backend developer. Our team was responsible for replacing a large part of an internal transaction processing system, and I’d been given ownership of several services. I was excited because it was the first time I’d had that much influence over an architecture.

I made a lot of decisions quickly.

I introduced asynchronous communication between services, pushed for a new event schema, and argued that we should separate several components before migrating the existing workflows. None of those ideas was inherently unreasonable, but I’d underestimated how many teams depended on the old system.

A senior engineer named Daniel warned me that our migration plan was too aggressive.

He wanted us to move one workflow first, measure the operational impact, and leave the rest of the system alone until we understood the failure scenarios. I thought he was being overly cautious. I’d spent weeks reading about distributed architectures, and I was convinced that keeping the existing design would create more problems later.

We went ahead with my approach.

Three months later, the project was behind schedule, and our staging environment was producing failures nobody could reproduce consistently. Some messages were processed twice, other workflows stalled when dependencies became unavailable, and our monitoring wasn’t detailed enough to explain what had happened. I started working longer hours. Then I started blaming everyone else.

When QA reported a bug, I complained that the test environment was unreliable. When another team questioned our integration contract, I said they hadn’t read the documentation. When Daniel suggested reducing the migration scope, I told him we were too far along to change direction.

Eventually, our Engineering Manager called me into a meeting. He said my technical work was good, but people were avoiding conversations with me. That hurt considerably more than being told I’d written bad code.

I Didn’t Fix It Immediately

Michael asked what I did after that meeting. I told him the embarrassing truth. For the first week, I convinced myself my manager had misunderstood the situation. I thought the team was frustrated because the project was difficult and that I’d become an easy target. I even considered leaving the company.

Then Daniel asked me to review a production issue with him.

We spent nearly two hours tracing a failed transaction across several services. He showed me how our new event processing flow had made a previously straightforward recovery procedure much harder. He wasn’t trying to prove he’d been right; he was trying to fix the system.

At one point, he asked why we’d separated a particular component. I started explaining the architecture. He stopped me and asked what problem the separation had solved for the business.

I couldn’t answer.

We’d introduced another service because I believed the architecture would be cleaner. In practice, we’d created an additional network dependency, more deployment coordination, and another place where transaction state could become inconsistent.

That was the first time I admitted the migration had become more complicated because of decisions I’d personally pushed for.

I told Daniel he’d been right about reducing the scope. He didn’t celebrate. He said we still had to finish the project.

The Interview Stopped Feeling Like an Interview

Michael asked how we’d recovered.

I explained that we stopped introducing new components and focused on stabilizing the workflows we’d already migrated. We improved our tracing, documented the failure states, and simplified several integration points. The project eventually shipped, although it took considerably longer than we’d planned.

Then he asked whether I’d repaired my relationship with Daniel.

I said mostly.

We worked together well afterward, but I don’t think he ever trusted my architecture recommendations quite as easily as he had before. I had to earn that trust back through smaller decisions, clearer explanations, and a willingness to change my mind.

Michael leaned back in his chair.

He said he’d experienced something similar when he became a tech lead.

For the next twenty minutes, we discussed the difference between being responsible for a technical decision and becoming emotionally attached to it. We talked about how difficult it can be to admit you’ve created unnecessary complexity, particularly when other engineers have already spent months implementing your proposal.

There was no whiteboard. Nobody asked me to design Twitter. I was having the first conversation in weeks where I didn’t feel like I needed to prove I was the smartest engineer in the room.

Then He Asked a Technical Question

With about twenty minutes remaining, Michael shared a small architecture diagram. It showed an order service, a PostgreSQL database, and a worker that sent requests to an external payment provider. Occasionally, the provider completed a payment but returned a timeout before the worker received confirmation.

He asked what I’d change.

Six weeks earlier, I probably would have started drawing a message queue and explaining retry strategies. This time, I asked how the system recorded payment attempts and whether the provider supported looking up an operation using a stable reference. Michael said the existing implementation marked timed-out payments as failed.

I explained why that was dangerous.

A timeout doesn’t necessarily mean the provider rejected the payment. The operation might have succeeded while the response was lost, leaving the application uncertain about the outcome. Retrying blindly could create duplicate charges if the provider didn’t enforce idempotency for that operation.

We discussed recording an explicit uncertain state and reconciling it before deciding whether another payment attempt was safe. I also explained that the exact recovery mechanism would depend on the provider’s API and the guarantees it offered.

Michael asked whether I’d ever implemented something like that. I said yes, although our first implementation had been worse than the one I’d just described. He laughed and asked what we’d gotten wrong.

I told him.

The Recruiter Called the Next Morning

I assumed the interview had gone well, but I’d thought that before. Several companies had seemed enthusiastic during interviews before rejecting me with generic feedback. I closed my laptop and tried not to spend the evening replaying every answer.

The recruiter called at 10:40 AM.

She said Michael wanted to move me to the final round, which would be a conversation with the Engineering Director. She also mentioned that he’d described our interview as one of the more useful technical discussions he’d had that month.

I asked whether he’d said anything about my system design answer. She said he’d specifically appreciated that I didn’t immediately start redesigning the entire system.

That made me smile.

The final interview happened four days later. The director asked about my experience leading technical projects, handling disagreements, and working with engineers who challenged my decisions. I used examples from the same migration, including the mistakes I’d made and the changes we’d introduced afterward. The company made an offer the following week. It wasn’t the highest salary I’d discussed during my job search, but the responsibilities were clear, and I liked the engineering team. I accepted after reviewing the compensation and on-call expectations.

What I Wish I’d Understood During Those First Eight Interviews

I spent most of my job search trying to convince companies that I was an exceptionally capable engineer. I prepared impressive technical explanations, polished my project stories, and practiced answering questions without hesitation. Somewhere along the way, I’d started treating uncertainty as something I needed to hide.

That made my interviews worse.

Experienced engineers know that projects go badly, architecture decisions fail, and people make mistakes. When someone describes a career where every difficult situation ends with them making the correct decision, the story can become difficult to believe. I hadn’t been lying about my experience, but I’d been leaving out the parts that explained how I’d actually developed judgment.

The ninth interview worked because I stopped editing those parts away. I still needed technical knowledge. Michael’s payment processing question required me to understand idempotency, transaction state, and failure recovery. A sincere conversation wouldn’t have compensated for an inability to reason about those problems.

But technical experience became easier to demonstrate when I wasn’t trying to turn every answer into a performance.

I think about that interview whenever I prepare for a difficult conversation now. The best explanation of an engineering decision usually includes the constraints, the alternatives, and the consequences that weren’t obvious at the beginning. Sometimes it also includes admitting that someone else understood the problem better.

I nearly canceled the interview because I thought it would be another wasted afternoon. Instead, it reminded me why I’d wanted to become a senior engineer in the first place.

Preparing for Your Next Senior Engineer Interview

If you’re going through a difficult interview process, I’ve put together a free Senior Engineer Interview & Production Playbook covering practical technical scenarios and production engineering concepts. For engineers preparing for advanced architecture discussions, my Senior & Staff System Design Interview OS goes deeper into distributed systems, reliability, and the decisions interviewers expect experienced engineers to explain.

I also write about software engineering careers, difficult technical decisions, and the parts of the job that rarely appear in interview preparation guides on Substack. If these experiences sound familiar, you’ll find more of them there.

I Did 9 Software Engineer Interviews in 6 Weeks. The One I Almost Canceled Changed Everything was originally published in Engineering Under Pressure on Medium, where people are continuing the conversation by highlighting and responding to this story.