01 · SCENARIO · Intermediate
Fictional teaching scenario. This is not an account of a particular deployment.
You are building an order service. After checkout it must (1) save the order in a database and (2) publish an OrderPlaced event to a queue consumed by the warehouse and notification services.
The team writes an INSERT, then calls publish() in the same service method. It works in a demo. In production, a customer sometimes gets confirmation but the warehouse never receives the order. In other cases the warehouse receives an event for an order that is absent from the database.
Your question
Why can this happen? How would you make the database change and event publication reliable? Why does swapping their order not solve it?
Guiding questions
- What if publish fails after a successful INSERT?
- What if the process dies between the two calls?
- Does swapping their order solve the problem or reverse the symptom?
- Does the database transaction cover the broker?