Systems work, described without the confidential details.

Real-time promotion personalisation

Make the next best promotion decision within the request budget.

A promotion system needs to decide what a customer is eligible to see while maintaining budget controls, business rules and experiment integrity. It also needs to do that at runtime, under high request volume, without allowing an unavailable dependency to block the product flow.

Problem & constraints

The problem was not simply matching a promotion to a user. The decision path had to evaluate eligibility and prioritisation with bounded latency, survive partial dependency failures and retain enough observability to explain why a decision happened. Every new check made the customer experience more precise—but also made the critical path more expensive and failure-prone.

Technical reasoning

The design favoured a deterministic, bounded decision flow: resolve the highest-value signals first, use carefully scoped cache reads for hot data and make the fallback behaviour an explicit product decision. Experiment assignment and budget-aware rules were considered part of correctness, not operational afterthoughts.

Trade-offs & outcome

The path deliberately trades some real-time enrichment for predictable latency and a more graceful failure mode. The result was a real-time personalisation platform that contributed to an approximately 3% improvement in order rate. Details of the individual rules and underlying services are intentionally withheld.

Reusable lesson

For a high-volume decision system, define what “correct enough during degradation” means before the incident. A consistent fallback is usually more valuable than a sophisticated decision that is only available on a healthy day.

Retry amplification & cascading failure

Retries are not a reliability feature unless the whole call graph can afford them.

A dependency under pressure invites the familiar response: retry. At one layer that may be sensible. Across a call graph, it can multiply work precisely when the system has the least capacity to do it.

Technical reasoning

The analysis starts with the complete request tree: client retries, gateway retries, RPC retries and datastore retries. A modest retry policy at every layer can inflate one customer action into many requests against a failing dependency. The control plane needs explicit ownership: one bounded retry policy, deadlines that propagate and a way to shed non-essential work.

Approach

The practical answer combines clear timeouts, bounded exponential backoff with jitter, retry budgets, concurrency limits and circuit-breaking or kill-switch controls where a product-safe fallback exists. Observability should reveal attempts and amplification, not only the final request outcome.

Reusable lesson

Design capacity for the failure path. If the only response to a slow dependency is more demand, the system is optimised to turn a local problem into a wider one.