Who does what when production breaks
“Walk me through how you handle a production issue. Who is involved?”
Three companies, three support lines, one leader. Stop the harm first, find the cause second, prove it cannot come back third.
See it
INC: quotes fail
runbook: logs, last deploy
1/8An alert fires and dealers call. Both become one ticket in the 's queue.
In plain words
When something breaks at BMW FS, three companies are involved. BMW FS owns the business: its () takes the calls, its IT manager approves changes, its team decides what customers are told, and its runs any SQL. FIS owns the core lending system; if the fault is inside it, a ticket goes to FIS support. We, the vendor, own the .NET services around it: our follows and flips switches, and the developers (L3) fix code. One person, the , leads, so the call does not turn into ten people guessing. Severity () decides how fast all this happens.
- Severity first: is customer or money harm right now with no workaround; it gets a within minutes and updates every 30 minutes.
- One runs the and the updates; the people debugging are not the people talking to the client.
- Mitigate before you diagnose: , flip a or scale out. The root cause can wait an hour; customers cannot.
- Then a : reproduce, find the root cause, fix with a test that failed first, release through a normal or .
- Close with a blameless : timeline, what allowed it, action items with owners. The client gets the .
What they ask next
Interviewers check that you can explain every line. Tap a question to open it.
What does the textbook say, and what really happens?
Textbook: an alert finds it, roles are clear, and the has the answer. Real life: customers often find it first, 's says "restart" (which can hide the bug), and the first hour is spent proving whose code it is: ours, FIS's or the network. A senior says what they know and what they don't in every update, and never guesses a cause to the client.
Rollback or : how do you decide?
Roll back when the trouble started with a release and the old version is safe: a takes seconds. Fix forward when the bug is older than the last release (rolling back changes nothing), or a database change cannot be undone. Either way, a that switches the feature off is often the fastest mitigation of all.
How do you decide or ?
Ask three things: how many users, is there money or legal harm, and is there a workaround. Every dealer blocked = . Wrong texts to some customers, with the texts switchable off = . One report screen slow = . When unsure, call it higher and lower it later; the reverse costs trust.
Whose code is it: ours or FIS's?
Follow one failing request in from end to end. If it fails inside our service, it is ours. If our call to the FIS core returns the error, we open a ticket with FIS support, attach the request id and timings, and keep a workaround running on our side.
What goes in the first update to the client?
What customers see, since when, how many are affected, what we are doing now, and the time of the next update. No guesses about the cause and no names. Compliance decides any wording that reaches customers.
Traps
- Don't say
I found the root cause first, then fixed it.SayWe mitigated first ( in minutes), then found the root cause calmly and shipped the fix through a change. - Don't say
The developer who wrote it made a mistake.SayBlameless: what let the bug through? A missing check, a test, a review rule. Those become action items. - Don't say
I restarted the service and it was fixed.SayA restart is a workaround. If it fixes it, state is living too long, so I open a .
AI helps you understand faster: drafts the , a chat tool explains a stack trace, and the chat can be summarised into a draft. It does not decide: severity, the call, the client wording and the cause stay human. Strip from any log before it goes into a chat tool.