Playbooks for incidents

CI playbook

Refer to the CI incidents playbook

Pausing a release

In some incidents, we may learn that we need to pause a release.

• The incident lead:
  • Alerts @delivery-support, @cs, @ce so the teams can plan accordingly
  • Handles the internal details of pausing a release (engaging the messenger to help with any relevant customer communication)
• The messenger alerts customers via a post on the status page

Go-to-market (license and subscription) issues

If a customer is experiencing an issue related to their license key or subscription status, any member of the Sourcegraph team has authority to generate a new, valid license key for any customer for any number of users that is valid for up to 7 days in the site-admin Subscriptions page on Sourcegraph.com. This will prevent the initial incident responder from being bottlenecked on a member of the go-to-market team that can validate the customer’s subscription status.

The incident responder will need to select a Sourcegraph.com account to attach the subscription to (typically the account should belong to the customer, so they can access the license key directly from their user profile, but in an emergency, the incident responder can use their own account in lieu of asking the customer), and can then manually generate a license key. No license “tags” are necessary.

If a customer’s instance is reporting “license expired” already, note that there is a 72hr grace period before non-admin users are locked out.

Figure out why a pod is in CrashLoopBackoff

CrashLoopBackoff indicates that a service is repeatedly failing to start. Instead of using Google Cloud Console to check the logs, the best way to figure out why a pod is in this state is to use the following commands:

kubectl get pods -o=wide --namespace=prod   # find the pod you are interested in
kubectl describe --namespace=prod pod $POD_NAME

In the output, you’ll find logs and a snippet like the following that can give more information about why a service has not started:

    State:         Waiting
      Reason:      CrashLoopBackOff
    Last State:    Terminated
      Reason:      OOMKilled
      Message:     690

Certain states like Terminated is not very clearly indicated in Google Cloud Console, and will usually not be properly indicated in logs either.

Restarting pods

This action is also referred to as “bouncing pods” or to “bounce a pod”. Follow the steps to authenticate kubectl and perform a restart using kubectl to restart the desired deployment:

kubectl rollout restart deployment/$SERVICE

Note the use of deployment/—you cannot restart services (svc/). Note that stateful sets have a different process.

You can also bounce pods by manually finding them via kubectl get pods and kubectl delete pod $POD_ID. Once a pod is deleted, it will automatically be recreated.

Making updates to stateful sets

Statefulsets are different to a deployment in that all pods must be in a healthy state before changes can be made.

Currently sourcegraph uses statefulsets for the following services:

• gitserver
• grafana
• indexed-search

In order to push an update to a failing statefulset take the following action:

kubectl apply -f <service.StatefulSet.yaml>

REPLICAS=`kubectl get sts -n prod <statefulset> -o jsonpath={.spec.replicas}`; for i in `seq 0 $[REPLICAS-1]`; do POD=<statefulset>-$i;   echo "Deleting POD $POD";   kubectl delete pod -n prod $POD ; done

watch -n1 kubectl get all -n prod -l app=gitserver -o wide

Useful dashboards

Free gitserver disk space on sourcegraph.com

kubectl get all -n prod -l app=gitserver -o wide

NAME              READY   STATUS            RESTARTS   AGE   IP             NODE                                NOMINATED NODE
pod/gitserver-0   0/1     CrashLoopBackOff  12         1h    10.52.14.6     gke-dot-com-dot-com-93160456-s7zw   <none>
pod/gitserver-1   1/1     Running           0          1h    10.52.13.189   gke-dot-com-dot-com-93160456-b7bk   <none>
pod/gitserver-2   1/1     Running           0          1h    10.52.0.165    gke-dot-com-dot-com-93160456-271w   <none>

kubectl -n prod describe pod gitserver-0 | grep -A5 "Message:"

      Message:   t=2019-04-24T07:46:46+0000 lvl=info msg="Distributed tracing enabled" tracer=Lightstep
2019/04/24 07:46:46 failed to setup temporary directory: mkdir /data/repos/.tmp-old227473847: no space left on device

gcloud compute ssh gke-dot-com-dot-com-93160456-s7zw --zone us-central1-f

sudo su
df -h | sort -r -k 5 -i | head -n8

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        3.9T  3.8T   68G  99% /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/repos-gitserver-0-dot-com
/dev/root       1.2G  466M  756M  39% /
overlayfs       1.0M  172K  852K  17% /etc
tmpfs           1.0M  144K  880K  15% /var/lib/cloud
overlay          95G  8.2G   87G   9% /var/lib/docker/overlay2/ed84d3f946264457738a05ba00d586e8de53e731983a21b6825faa6fef01f068/merged
overlay          95G  8.2G   87G   9% /var/lib/docker/overlay2/d7ca574132a2511a82b3bbaa0010ee49e7e616a1ebee9ca61ad094d4aef286dd/merged
overlay          95G  8.2G   87G   9% /var/lib/docker/overlay2/d7bd7cc72a2e47cb32f2542bd7416597d4ad7a6409d0c069b041bf28766dea2a/merged

cd /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/$DISK_NAME/github.com
for repo in *; do echo $repo; rm -rf $repo; sleep 0.1; df -h .; done

watch -n1 kubectl get all -n prod -l app=gitserver -o wide

PostgreSQL database problems

We provide two sets of instructions here, shell commands and PostgreSQL commands to be run inside a psql instance. PostgreSQL commands are denoted by the prompt sg=# in this documentation; the actual prompt corresponds to the postgres user name.

alias dbpod='kubectl get pods --namespace=prod | grep pgsql | cut -d " " -f 1'
alias proddb='kubectl exec -it --namespace=prod $(dbpod) -- psql -U sg -P pager=off';

kubectl exec -it --namespace=prod $(dbpod) -- top

sourcegraph=# select * from schema_migrations;
  version   | dirty
------------+-------
 1528395540 | f
(1 row)

sourcegraph=# \d schema_migrations
Table "public.schema_migrations"
 Column  |  Type   | Modifiers
---------+---------+-----------
 version | bigint  | not null
 dirty   | boolean | not null
Indexes:
    "schema_migrations_pkey" PRIMARY KEY, btree (version)

           v-- this column here is really useful!
sourcegraph=# select (version, dirty
sourcegraph(# )
sourcegraph-# from schema_migrations
sourcegraph-# ;
      row
----------------
 (1528395540,f)
(1 row)

select pid, now() - pg_stat_activity.query_start AS duration, query, state from pg_stat_activity where (now() - pg_stat_activity.query_start) > interval '1 minute';

select pg_cancel_backend(pid), now() - pg_stat_activity.query_start AS duration, query, state from pg_stat_activity where (now() - pg_stat_activity.query_start) > interval '1 minute';

select pg_terminate_backend(pid), now() - pg_stat_activity.query_start AS duration, query, state from pg_stat_activity where (now() - pg_stat_activity.query_start) > interval '1 minute';

SELECT * FROM schema_migrations;

  version   | dirty
------------+-------
 1528395539 | t
(1 row)

UPDATE schema_migrations SET dirty = 'f' WHERE version = 1528395539;

sg=# \d global_dep
   Table "public.global_dep"
  Column  |  Type   | Modifiers
----------+---------+-----------
 language | text    | not null
 dep_data | jsonb   | not null
 repo_id  | integer | not null
 hints    | jsonb   |
Indexes:
    "global_dep_idx_package" btree ((dep_data ->> ('package'::text COLLATE "C")))
    "global_dep_idxgin" gin (dep_data jsonb_path_ops)
    "global_dep_language" btree (language)
    "global_dep_repo_id" btree (repo_id)
Foreign-key constraints:
    "global_dep_repo_id" FOREIGN KEY (repo_id) REFERENCES repo(id) ON DELETE RESTRICT

SELECT
  now()::TIME(0),
  a.query,
  p.phase,
  --round(p.blocks_done / p.blocks_total::numeric * 100, 2) AS "% done",
  p.blocks_total,
  p.blocks_done,
  p.tuples_total,
  p.tuples_done,
  p.lockers_done,
  p.lockers_total,
  p.current_locker_pid,
  ai.schemaname,
  ai.relname,
  ai.indexrelname
FROM pg_stat_progress_create_index p
JOIN pg_stat_activity a ON p.pid = a.pid
LEFT JOIN pg_stat_all_indexes ai on ai.relid = p.relid AND ai.indexrelid = p.index_relid;

   SELECT blocked_locks.pid     AS blocked_pid,
         blocked_activity.usename  AS blocked_user,
         blocking_locks.pid     AS blocking_pid,
         blocking_activity.usename AS blocking_user,
         blocked_activity.query    AS blocked_statement,
         blocking_activity.query   AS current_statement_in_blocking_process,
         blocked_activity.application_name AS blocked_application,
         blocking_activity.application_name AS blocking_application
   FROM  pg_catalog.pg_locks         blocked_locks
    JOIN pg_catalog.pg_stat_activity blocked_activity  ON blocked_activity.pid = blocked_locks.pid
    JOIN pg_catalog.pg_locks         blocking_locks
        ON blocking_locks.locktype = blocked_locks.locktype
        AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
        AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
        AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
        AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
        AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
        AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
        AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
        AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
        AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
        AND blocking_locks.pid != blocked_locks.pid

    JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
   WHERE NOT blocked_locks.GRANTED;

$ kubectl -n prod get pods | grep redis
redis-cache-8475866c76-swxlz                         2/2     Running     0          22d
redis-store-7c5fb8dd89-jzx44                         2/2     Running     0          22d

$ kubectl -n prod exec -it redis-store-7c5fb8dd89-jzx44 -- redis-cli
Defaulting container name to redis-store.
Use 'kubectl describe pod/redis-store-7c5fb8dd89-jzx44 -n prod' to see all of the containers in this pod.
127.0.0.1:6379>

DNS/SSL settings on Cloudflare

• Visit https://dash.cloudflare.com/login (get the credentials from 1Password)
• Click on “sourcegraph.com”
• Click on DNS and make sure the IP matches the IP from kubectl get -nprod services (after settings the K8s context with gcloud container clusters get-credentials main-cluster-5 --zone us-central1-f --project sourcegraph-dev)
• Click on Crypto in the top bar and make sure it’s set to Full

Rolling back Sourcegraph.com to an old frontend image version

• Run gcloud container clusters get-credentials main-cluster-5 --zone us-central1-f --project sourcegraph-dev
• Visit https://hub.docker.com/r/sourcegraph/frontend/tags/ to find an older tag
• Run kubectl edit deployment -nprod sourcegraph-frontend and find the line that looks like image: sourcegraph/frontend:22342_2018-10-30_a0ba521 and update the tag

Deleting a repository and its data

1. Get pod names in production: kubectl --namespace=prod get pods
2. Connect to Postgres: kubectl --namespace=prod exec -ti [pgsql pod name] -- psql -U sg
3. Delete row: DELETE FROM repo WHERE uri = 'github.com/example/example';
4. Connect to gitserver-1: kubectl --namespace=prod exec -ti [gitserver-1 pod name] -- sh
5. Find and delete repo directory in /data/repos/ if exists
6. Do steps 4 and 5 with gitserver-2 as well

Failing about.sourcegraph.com

Several of our static sites are hosted by Netlify at about.sourcegraph.com. You can check the service status at https://www.netlifystatus.com/ as well as notifications to the #alerts-external channel.

If you discover an issue with Netlify but their status page is not displaying it, open a support request using the credentials in 1Password.

Sites:

• about.sourcegraph.com
• sourcegraph.com/about -307-> about.sourcegraph.com/about
• sourcegraph.com/pricing -307-> about.sourcegraph.com/pricing

For a full list of sites redirected from sourcegraph.com/$KEY reference the router code in our frontend.

Moving a GCE disk between projects

Moving a disk between projects requires multiple steps to ensure the disk is detached from its original source.

• The initial disk can only be created using the same type. To change its type, you will have to snapshot the disk and restore as a new type.
• The initial disks can only be moved to the same region as their source project.

Create a new disk in your target project from the old disk:

gcloud compute disks create ${TARGET_DISK_NAME} --source-disk=https://www.googleapis.com/compute/v1/projects/${SOURCE_PROJECT}/zones/${SOURCE_ZONE}/disks/${PVC_NAME} --project=${TARGET_PROJECT} --zone=${TARGET_ZONE} --type ${SOURCE_TYPE}

We now have to snapshot and restore the disk, to ensure the disk is unlinked from its source disk. This process is the same as its used to change a disk zone or type, please follow that playbook.

Changing a GCE disk zone or type

To change disk region or type, we need to first create a snapshot and then restore it to the new region or type.

Snapshot:

gcloud compute disks snapshot ${SOURCE_DISK_NAME} --project=${PROJECT} --zone=${SOURCE_ZONE} --snapshot-names ${SNAPSHOT_NAME}

At this point, you can optionally delete the source disk.

Restore:

gcloud compute disks create ${TARGET_DISK_NAME} --project=${PROJECT} --size ${TARGET_DISK_SIZE} --source-snapshot ${SNAPSHOT_NAME} --zone=${TARGET_ZONE} --type ${TARGET_DISK_TYPE}

TARGET_DISK_SIZE has to be equal or larger than the original source disk.

There is an alias command that performs the snapshot -> restore operation automatically to switch regions, but it fails on big disks.

Changing Persistent Volume disk or other immutable parameters (PV replacement)

As changing i.e. gcePersistenDisk or most PVs parameters are not allowed, steps to replace the PV with minimal downtime period:

1. Prepare new disk - use Changing a GCE disk zone or type paragraph above.

2. Create new PV yaml file (sample here)

3. Apply new changes via opening, approving and merging a PR.

4. Check if new PV is available - kubectl get pv -n prod.

5. Prepare PR with old PV removal, which should be deleted in the base folder.

6. Delete PV and PVC which should be replaced - via kubectl.

kubectl delete pv <PV-to-be-removed>
kubectl delete pvc <PVC-to-be-replaced> -n prod

7. Merge PR from step 5 - this will invoke a deploy again, to recreate the PVC via a buildkite job.

8. Verify that new PV is used (should be in Bound status):

kubectl get pv

Linking a PV/PVC to a GCE disk

When we need to re-link a disk to a PersistentVolume and its associated PersistentVolumeClaim because we are restoring a disks from a snapshot or for some other reasons, we will need to adapt the Kubernetes resource definitions to link the existing GCE disk to GKE.

Example PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: MyDisk
  namespace: SomeNamespace
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

We will first ensure that the Claim is directly linked to an fixed Persistent Volume by setting the claimRef to the desired PVC. You also need to ensure that the accessModes, capacity and all other settings match between the claim and the volume.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: MyDisk
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    name: MyDisk
    namespace: SomeNamespace
  storageClassName: SomeStorageClass
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: MyDisk
  namespace: SomeNamespace
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Lastly, we need to link the volume to a GCE disk by setting gcePersistentDisk. We must ensure that gcePersistentDisk.pdName matches the disk name in GCE.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: MyDisk
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    name: MyDisk
    namespace: SomeNamespace
  gcePersistentDisk:
    fsType: ext4
    pdName: SomeGCEDisk
  storageClassName: SomeStorageClass
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: MyDisk
  namespace: SomeNamespace
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: MySet
  namespace: SomeNamespace
[...]
spec:
  [...]
  replicas: 1
  volumeClaimTemplates:
    - metadata:
        name: DataDisk
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: SomeStorageClass

apiVersion: v1
kind: PersistentVolume
metadata:
  name: DataDisk-0
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    name: DataDisk-MySet-0
    namespace: SomeNamespace
  gcePersistentDisk:
    fsType: ext4
    pdName: SomeGCEDisk
  storageClassName: SomeStorageClass

Export/Import a Cloud SQL database

Exporting/Importing a database to Cloud SQL using gcloud requires the export to existing in a Google Storage bucket and it can be performed across GCP projects.

Export:

gcloud --project=${SOURCE_PROJECT} sql export sql ${SOURCE_SERVER_NAME} 'gs://some-bucket/Cloud_SQL_Export_${SQL_SERVER_NAME}_${DB_NAME}' --database=${SOURCE_DB_NAME}

Import:

gcloud --project=${TARGET_PROJECT} sql import sql ${TARGET_SERVER_NAME} 'gs://some-bucket/Cloud_SQL_Export_${SOURCE_SERVER_NAME}_${SOURCE_DB_NAME}' --database=${TARGET_DB_NAME}

Sourcegraph.com is deleted entirely

This playbook has a dedicated page.