Delete a Cloud instance

Prereq

Follow https://github.com/sourcegraph/controller#installation to install mi2

git clone https://github.com/sourcegraph/cloud
cd cloud

Install mi2 binary

go install ./cmd/mi2/

Steps

See flow chart https://app.excalidraw.com/s/4Dr1S6qmmY7/2wUSD4kIxRo

  1. Set environment variables
  2. Check out a new branch
  3. Modify instance config to use TFC cli mode
  4. Remove namespace
  5. Disable delete protection
  6. Remove GKE backups and restores
  7. Destroy infrastructure - destroy cdktf stacks
  8. Delete TFC workspaces
  9. Commit your changes

Set environment variables

Sharing TF_TOKEN_app_terraform_io is only temporary, this is expected to change in the future.

Bash

export SLUG=company
export DOMAIN=company.sourcegraph.com
export ENVIRONMENT=dev
export TF_TOKEN_app_terraform_io=$(gcloud secrets versions access latest --project=sourcegraph-secrets --secret=TFC_TEAM_TOKEN)
export INSTANCE_ID=$(mi2 instance get -e $ENVIRONMENT --slug $SLUG | jq -r '.metadata.name')

Fish

set -x SLUG company
set -x DOMAIN company.sourcegraph.com
set -x ENVIRONMENT dev
set -x TF_TOKEN_app_terraform_io (gcloud secrets versions access latest --project=sourcegraph-secrets --secret=TFC_TEAM_TOKEN)
set -x INSTANCE_ID (mi2 instance get -e $ENVIRONMENT --slug $SLUG | jq -r '.metadata.name')

Check out a new branch

git checkout -b $SLUG/delete-instance

Modify instance config to use TFC cli mode

Change the Terraform Cloud run mode to CLI-driven Note: this will remove VCS trigger from Terraform Cloud workspaces for this instance!

mi2 instance edit --query '.spec.debug.tfcRunsMode = "cli"' --slug $SLUG -e $ENVIRONMENT
cd environments/$ENVIRONMENT/deployments/$INSTANCE_ID/
npx --yes cdktf-cli@0.13.3 deploy tfc

Remove namespace

mi2 instance workon -e $ENVIRONMENT --slug $SLUG
#opy and run the output `gcloud` and `kubectl` commands

# deletes namespace and Network Endpoint Group Health check
kubectl delete ns $NAMESPACE

Disable delete protection

# delete sql protection
cd environments/$ENVIRONMENT/deployments/$INSTANCE_ID/terraform/stacks/sql
terraform init
export SQL_RESOURCE=$(terraform state list | grep sql_self)
echo "$(jq '.resource.'$SQL_RESOURCE' += {"delete_protection":false}' cdk.tf.json)" > cdk.tf.json
terraform apply -auto-approve

Removes GKE backups and restores

# remove GKE restores, backups, restore-plans and backup plans
cd sourcegraph/cloud
export PROJECT_ID=$(mi2 instance get -e $ENVIRONMENT --slug $SLUG | jq -r '.status.gcpProjectId')
gcloud config set project --project $PROJECT_ID
gcloud beta container backup-restore restores list | awk '{print $1}' | xargs gcloud beta container backup-restore restores delete
gcloud beta container backup-restore backups list | awk '{print $1}' | xargs gcloud beta container backup-restore backups delete
gcloud beta container backup-restore restore-plans list | awk '{print $1}' | xargs gcloud beta container backup-restore restore-plans delete --async
gcloud beta container backup-restore backup-plans list | awk '{print $1}' | xargs gcloud beta container backup-restore backup-plans delete --async

Destroy infrastructure - destroy cdktf stacks

the stack list may be out-of-date, run npx --yes cdktf-cli@0.13.3 under the instance root in case things are not working as intented

cd environments/$ENVIRONMENT/deployments/$INSTANCE_ID/
npx --yes cdktf-cli@0.13.3 destroy project network gke sql app sqlschema waf security executors monitoring output --auto-approve --parallelism 8

If previous step fails for any reason, fallback to pure terraform destroy:

cd environments/$ENVIRONMENT/deployments/$INSTANCE_ID/terraform/stacks/
for stack in output monitoring executors security waf app sqlschema sql gke network project; do cd $stack && terraform init && terraform destroy && cd ..; done

Delete TFC workspaces

cd environments/$ENVIRONMENT/deployments/$INSTANCE_ID/
npx --yes cdktf-cli@0.13.3 destroy tfc

Commit your changes

rm -rf environments/$ENVIRONMENT/deployments/$INSTANCE_ID
git add .
git commit -m "$SLUG: delete instance"

Create a new pull request and merge it

Any other questions?

Please reach out to #cloud