Serving as a messenger during incidents

You are here because our engineering teammates need a messenger as they work on addressing an incident. And you were either asked by CS leadership or proactively jumped in because you saw a request for messenger in #incidents, #customer-support, or some place else.

The messenger is always a member of customer support (whether an application engineer, program manager, or leader) and they are responsible for customer-facing communication about/during an incident and coordinating the necessary folks internally to create/sign-off on our communications. Having support be the team to send these communications allows engineering to focus on fixing the issue and customer engineering to be a point of escalation as customers process what an incident may mean for them. It also more easily allows us to keep Zendesk a system of record.

Our responsibilities as the messenger are to…

• Remain calm.
• Ensure we understand impact and validate the necessity of customer communication.
• Ensure we have complete and accurate details to share with our customers.
• Coordinate relevant folks internally to create/sign-off on/be aware of the communication plan.
• Show up for our teammates involved with kindness and perspective … no matter how bad an incident may feel, it’s not grave, and it’s about coming together for what’s next more than how we got here (we retro after an incident is handled).
• Work quickly with attention to detail and an intense focus on the customer experience, remembering speed matters as much as how customers may feel through the process.

To serve as a messenger successfully, follow the steps outlined here in the exact order listed. Sure some things may happen out of order, but go through it all in this order … Doing so will allow you to show up for our teammates and customers with calm and organization.

1. Confirm in all of the places necessary that you are on it and state that you need a few minutes to catch up on this. Typically, you’ll confirm this in a thread in #customer-support and/or a thread in #customer-support-internal and/or a thread in #incidents and or the #inc channel created for this particular incident. The point here is to just let folks know you are on it, don’t do anything yet … proceed with the steps as outlined.
2. Count to 10.
3. Take a deep breath and let that breath out.
4. Take a sip of water.
5. Post in #customer-support-internal, @ mention @cs, and ask for a member of the team (it can be anyone … an AER, a program manager, or a leader) to serve as your buddy. As you get caught up in the energy of the incident, their job is to maintain distance and double check with you that you have done all of the steps in this precise order and serve as a copy editor on the communications.
6. Together with your buddy, read the rest of the steps before starting step 7. No matter how many incidents for which you serve as messenger (or buddy to the messenger), the next 5-8 minutes will go far in helping you remain calm. When you are done reading, then proceed to do the next step. If at any point you feel unsure or stuck, that’s okay! Incidents can be unsettling, chaotic, and hard to navigate. Just engage CS leadership and they will show up for you and help however you need.
7. Create a folder for the incident here either with the inc-## number and a descriptive name or just a descriptive name if there is not yet a inc-## number (we can change it to the inc-## number later)
8. Make a copy of the incident communication plan template [TEMPLATE] Inc-## communication plan and save it in the folder (renaming it to reflect this incident) and filling in the details at the top in yellow highlight.
9. Read the incident thread and channel and start filling in the details in the gather information section of the communication plan. If you are not seeing these details as you read the threads/channel, ask in the channel (or the thread if there isn’t a channel yet). As you read up, if it’s a security incident, STOP!

   • It is critical that we follow [our security disclosure process](../../../admin-exp/security/reporting-vulnerabilities.md#how-we-disclose-security-vulnerabilities). Is a security engineer already involved? If not, post in #security and @ mention @security-team and ask who is available to assess an incident and lead from the security side. Itt is critical that we act in deference to the security team from this point forward in terms of assessing the issue, impact, severity, etc. You will still gather information, but we need to be sure it’s coming with security sign-off.
     

   • Now, get a member of the CS leadership team. We want to be extra careful during security incidents since these can hurt our reputation the most. Post in #customer-support-internal and @ mention @cs-leadership and let the leadership team know you need one of them for eyes on a security incident. Their role will be to double check communications and think through business implications. Normally we wait for the communications review to involve CS leadership, but for security incidents, the earlier the better. You can use the same thread you created to get a buddy or a new one – your choice!
10. Generate the impacted customer contact list and link it in the designated section of the communication plan. Your end goal of this step is to have a Google sheet that you can link here that lists which customers/users need to be informed and has the related email address, in addition to the relevant CE and AE for each customer. This list comes from two primary sources:

    • Either engineering provides a list of customers/users
      

    • Or, we have to manually back into a list based on key data. If this needs to be the source, first generate a Salesforce report that lists all customers and shows whether their CE has listed incident contacts. If possible, mark customers as impacted or not yourself. If you need CEs to help identify, post in #ce and ask for their help using this template (be sure to prep the Google Sheet first). You may need to follow-up with CEs individually to be sure you end up with a complete and accurate list.
    • Note: As of 2022-01-14 this is the desired future state (having a list in Salesforce). Check-in with Aimee on the best way to start this if they haven’t already shared this with you.
11. Based on the details you have gathered and using these templates, create the draft communications in the designated section of the communication plan.
12. Fill out the timeline in the communication plan with planned dates. Incident work can span multiple days. In some cases we may have a legal requirement for disclosure of 72 hours. Either way, we want to be expedient but ensure we have accurate information. Generally, planned dates might span today’s date and tomorrow so there is enough time to let folks review communications, prepare the impacted customer list, etc. If an event is not necessary, mark the planned and completed dates as n/a (not applicable).
13. As you continue with the next steps, fill out the timeline in the communication plan with completed dates and relevant links (to Slack threads, the status page post, etc). You want that timeline table to be a quick reference overview that collates your work as you go.
14. Solicit review of the draft customer communications from your buddy, CS leadership, the incident lead, security lead (if relevant), marketing, legal (if relevant – this is decided between you and CS leadership), and simultaneously inform your CS teammates and the CE team using these templates. Note that you will put forth a recommendation to CS leadership on whether to use the status page, so think through that for that review request.
    • Seek review from your buddy and CS leadership via a post in #customer-support-internal
    • Seek review from the incident lead and (if relevant) the security lead in the incident channel/thread
    • Seek review from marketing by posting in #marketing and @ mentioning @marketing-incident, @amie
    • Seek review from legal (if agreed this is necessary with CS leadership) by posting in #legal and @ mentioning @legal-team
    • Inform your CE teammates by posting in #ce
15. Incorporate edits and finalize the customer communications. There may be lots of comments and changes. That’s okay! The template was just a starting point and this is the step where we capture all of the nuance – what if this is the 3rd incident for this customer, etc.
16. Post/send customer communications.

    • We don’t always post to the status page. For example, we will never post about a security incident on the status page. During a security incident, it’s critical that we limit information so as to not create avenues of awareness that could make the situation worse. We also try to limit the status page for wide-reach issues, when the majority (somewhere between 50% and 80%) of self-hosted or Cloud customers are impacted. When we do post to the status page, we often post just one update as “resolved,” but we have templates ready [here](#status-page) for if we need to post a sequence of updates (if we do go with a sequence, we have two posts for “investigating” that can be posted after each 30 minutes until both are posted). [Here](#how-to-update-the-status-page) are the instructions for updating the status page.
      

    • In order to best honor our customer-first value, we take the time and care to reach out to customers in a direct one-to-one email. You will send these emails from your work Gmail (not Zendesk) with the relevant CE, relevant AE, and support@sourcegraph.com in copy. While this takes more time, it ensures emails don’t get lost in spam and helps maintain trust with the customer who will appreciate this level of care. It is also very important to remember to copy support@sourcegraph.com so that you can work from Zendesk for any follow-up with customers.
17. Help triage put all of the cases generated by sending emails into your queue and prep them for customer follow-up. Change the requestor to the customer and set the case to pending so it’s easy to tell if a customer replies (the case will flip to open).
18. Double check that you have filled out the timeline to reflect when things were completed and add relevant links (ie, add links to Slack posts, status page posts, etc). Again, you want that timeline table to be a quick reference overview that collates your work as you go.
19. It’s probably best that you be super careful of taking cases for the next few days so you can focus on being super responsive to customers who require follow-up.
20. Keep a sharp eye out for any follow-up responses/questions from customers. Post questions in the relevant inc-## channel and work with your engineering teammates to confirm the correct answer before answering the customer’s question. Capture the questions and answers in the FAQ section of the communication plan.
21. Make note of customers who would like you to send them our post-mortem in the relevant section of the communication plan and be sure to follow through on those requests. It can take a couple of weeks sometimes for our product teammate for that part of the product to complete the post-mortem, but that is usually who makes sure they get done. If a customer wants it, be sure to let the relevant PM and incident lead know so they can commence work sooner than later.
22. After 3 days, you can close all tickets that don’t require follow-up. If a customer writes back later, it will open a new ticket so nothing will fall through the cracks.
23. Either to create the customer-facing post mortem or after, engineering/product also lead a retro. Participate! Don’t try to change the process in the middle of an incident. Make notes and save it all for the retro. Changes are good and healthy, but the timing has to be right. And mid-incident changes create unnecessary confusion and stress … and remember, we want to show up and bring calm. We definitely want to iterate and improve, too, just with super thoughtful timing.
24. Celebrate! You have successfully served as a messenger on an incident and your work was integral to helping us maintain trust with our customers!