🗞 Developer experience newsletter

Tracing with OpenTelemetry

OpenTelemetry is now the default tracing export mechanism in Sourcegraph.

This means that traces now export to an OpenTelemetry Collector instance, which can then easily be configured to export data to a variety of backends - for example, in s2 we are currently exploring exporting to both Honeycomb, Google Cloud Traces, and Jaeger to assess the available options.

OpenTracing API calls are bridged to OpenTelemetry automatically, though we strongly recommend that everyone migrate to OpenTelemetry APIs (either internal/trace or go.opentelemetry.io/otel), which brings better API ergonomics and improved usage of context.Context. We’ve added a linter that forbids new imports of OpenTracing APIs, and added deprecation notices on internal APIs that should no longer be used.

Additionally, the web app can now be instrumented with OpenTelemetry as well, with traces being sent to the frontend service which proxies it to the deployed OpenTelemetry Collector.

To get started with testing out OpenTelemetry locally, refer to our OpenTelemetry development docs and refer to our tracing for site admins guidance.

If you are interested in learning more about OpenTelemetry in general and the specifics of the engineering work required to make this happen, check out this recording of a DevX team Q&A about OpenTelemetry!

Frontend news

esbuild for faster frontend builds: Shoutout to Nick Snyder who took it upon himself to get esbuild in a usable state for Sourcegraph. It doesn’t work for everything yet but a few people have reported a markable improvement in sg start startup time! You can enable it by running DEV_WEB_BUILDER=esbuild sg start. For more information on esbuild please see the following docs.

Documentation for analyzing the bundlesize check failure: With more contributions focused on the core workflow improvements, the Frontend Platform team noticed an increased number of bundlesize check failures, and often the reason for it is not apparent. Valery added a step-by-step guide for debugging root causes which should help teams to triage this failure. The Frontend Platform team will be looking into automating these steps entirely next quarter!

sg goodies

Finding builds or logs by commit: Ever wanted to look at the build of a particular commit? Not in the mood to go through all the pages of Buildkite? You don’t have to anymore! With sg ci status you can now pass it a --commit flag and it will do all the detective work for you. sg ci logs also accepts the --commit flag so you can now easily look at the logs of a build for a particular commit too!

Build annotations, in your terminal!: Ever wanted to check the status of your build in your terminal, but couldn’t see those nice little dialogues (annotations) at the top your build showing that test that failed and other helpful links? Well those days are gone! When you check the status of your build with sg ci status it will now also print any annotation that is present on your build!

Inspect main branch tags with sg ops inspect-tag: We’ve added a new subcommand named inspect-tag which allows you to inspect main branch tags. For example you can now inspect the image with sg ops inspect-tag index.docker.io/sourcegraph/cadvisor:159625_2022-07-11_225c8ae162cc@sha256:foobar or get the build number with sg ops inspect-tag -p build 159625_2022-07-11_225c8ae162cc. For more examples and other options see sg ops inspect-tag --help.

Update images in Docker compose manifests with sg ops update-images: update-images has been updated and can now update docker compose manifests with sg ops update-images -k compose. With compose entering the fold, sg ops update-images is now able to update images in three different formats namely k8s, helm and compose.

Commands in sg.config.overwrite.yml should no longer cause as much headache: Thorsten has fixed an issue that many have felt the pain of! We’ve all added a custom command in sg.config.overwrite.yml only for go run generate to come along and ruin our dreams. Thorsten, having been bitten by this one too many times, landed a fix for this by adding a flag -disable-overwrite to sg which is passed to sg when go run generate runs to generate the reference.md file. Due to the nature of the fix, there is nothing for you to do!

Multi user Auth testing just got a whole lot easier: Keegan recently added a http-header auth-proxy that creates a few users and exposes each user on a different port locally. By accessing Sourcegraph through these ports you are authenticated as that user. To use it run go run dev/internal/cmd/auth-proxy-http-header.go. Words don’t do it justice so below is some output of it in action! For more information on how authentication happens please see the docs.

go run auth-proxy-http-header.go
https://docs.sourcegraph.com/admin/auth#http-authentication-proxies

  "auth.providers": [
    {
      "type": "http-header",
      "usernameHeader": "X-Forwarded-User",
      "emailHeader": "X-Forwarded-Email"
    }
  ]

Visit http://127.0.0.1:10810 for william william@sourcegraph.com
Visit http://127.0.0.1:10811 for user1 william+user1@sourcegraph.com
Visit http://127.0.0.1:10812 for user2 william+user2@sourcegraph.com
Visit http://127.0.0.1:10813 for user3 william+user3@sourcegraph.com
Visit http://127.0.0.1:10814 for user4 william+user4@sourcegraph.com
Visit http://127.0.0.1:10815 for user5 william+user5@sourcegraph.com

A complete set of commands to run web application Puppeteer tests: Have you ever wondered how to debug a CI client flake locally? Always unsure of what environment variables to set to get things right? sg got your back! Valery updated respective commands, which now link directly to the up-to-date documentation:

sg test web-e2e
sg test web-regression
sg test web-integration
sg test web-integration:debug PATH_TO_THE_TEST_FILE_TO_DEBUG

CI improvements

Go to the Grafana logs of your build straight from your build: Previously, if you wanted to see the logs of your build you had to navigate to http://sourcegraph.grafana.net and wield the dark arts of creating a LogQL yourself to query the logs. We’ve updated annotations on builds to have an additional link named “View Grafana logs” which will take you directly to Grafana with a prefilled LogQL query for your particular build. One small step to helping you diagnose build failures in your faster!

Flaky tests happen, which is why the docs have a specific section about how to deal with those. In a nutshell, disable them on sight and notify the ownning team so they can fix those. We want to say thank you those who took a few minutes out of their day to improve the CI experience for everyone else. Special thanks to Thorsten Ball who disabled the most of them on his own! And thanks to Camden Cheek, Alex Ostrikov who are following closely.

All builds on the main branch are now faster by about 6 to 8 minutes. This is achieved by caching the client bundle build in the job that builds the server container that is later used to run e2e tests.

The caching mechanism is disabled on releases, to be 100% sure we are shipping the right client bundle. When the caching is used, an annotation such as the one below is displayed, making it explicit from when the client bundle was cached, so we can easily see if it should have been invalidated. Those two extra precautions are taken because the caching is done externally and do not rely on any client tooling, therefore invalidation depends on how careful we are at not missing files that could change the build result.

Stability improvement for frontend steps: Npm has became increasingly unstable over the past weeks, which caused an increase in client flakes when fetching packages that our code depends on. The CI now wraps yarn install in a retrying loop to mitigate those. Oldest trick in the book!

Build notifications: We’ve rolled out new build notifications! What was wrong with the old ones you might ask? Well, they just took you to the build and you were on your own from then on. With the new build notifications, we show what job failed on your build! We provide a link for you to go straight to the jobs output! If that wasn’t enough, we’ve also added a way for you to see all the logs of your build in Grafana. All of this from the comfort of Slack. We’ve aimed to make the notifications more actionable and since we’re now in control of the notifications we aim to make more improvements.

More annotations!: We’ve added a custom Mocha reporter which will upon any E2E and QA test failures generate a annotation on Buildkite with the relevant failures. Now you don’t have to doom scroll through a bunch of lines just to find the failures! The default buildkite failures Slack messages are not the most actionnable ones, so we have decided to replace them with our in-house notifications which we have total control on:

S2 news

Sentry frontend and backend errors are now available in Sentry on the S2 project. It has a much lower volume than dotcom and reflects more accurately what we can expect to see in our customers deployments. Therefore, it’s a prime candidate to monitor your own domain and to create alerts for issues relevant to your team, thanks to the scope attribute. You can view the loom recording to refresh your memory about logging scopes.

Continuous deployments are live: every hour, S2 is updated with the latest known green commit from the main branch.

New guides

Logging: We’ve made some updates to our logging guidance around the new logging library to include more concrete suggestions and examples around:

• How to use scopes
• Creating sub-loggers (e.g. those with traces and fields)
• Writing log messages

You can find the new docs in How to add logging!

Investigating flakes in CI: Have you ever merged a PR, got pinged in #buildkite-main, and thought “gosh this test failure has nothing to do with my changes 😭“? Well with a few quick steps you can easily determine if you’ve been hit with a frequently flaking test that should be disabled ASAP, and contribute to keeping our pipelines healthy! Check out our new Loom demo to learn more:

More details are available in the handbook: Grafana Cloud - CI logs, and if you have any questions please reach out in #dev-experience!

Observability features

Sentry errors: Errors are now automatically reported to Sentry from warning-level and above logs entries from sourcegraph/log loggers that include an error field, which can be added using the log.Error or log.NamedError field constructors, for example:

logger.Error("something terrible happened", log.Error(err))

The reported Sentry event includes helpful information like the logger scope, log message, log fields, and more! Check out this demo to see it in action:

Because this raises the amount of errors being reported, we’re experimenting with sampling the errors, which both prevents to fill our quota too quickly and also ensure that we’re not spending too much resources on the reporting itself.

Under the hood, it uses a new “log sinks” mechanism that can easily be extended to accomodate new backends in the future - you can learn more about it in the package docs! As a byproduct, the codebase doesn’t have anymore any explicit reference to Sentry, apart from the optional Sentry sink itself.

DataDog sunset: Note that the DataDog trial is being ended, and the plan is to sunset DataDog integrations within the codebase.

CLA bot automation

The process of ensuring an external contributor has signed Sourcegraph’s Contributor License Agreement (CLA) is now fully automated. Once a contributor has signed the CLA form, their provided GitHub handle will now automatically be synchronized to the list of approved users.

You can learn more in the clabot-config repository and the accepting external contributions guide.

sg goodies

sg [cmd...] --feedback: Love (or hate) sg? Want to make a suggestion or found that a command was acting strange? In addition to the --feedback flag on all commands, we’ve also added a feedback subcommand enabling you to give us feedback right from the comfort of your terminal! When you opt to provide feedback a new discussion will be created in the dev experience category on the Sourcegraph repository.

Generated sg documentation: Because maintaining documentation is always hard, what’s better than automation to make sure its stays up to date? The sg reference is now automatically generated.

sg setup and sg lint rewrite: We have rewritten sg lint and sg setup to use a shared internal framework that can be used to easily and flexibly build powerful “check-and-fix” tasks. This enables features like:

• sg setup -check to quickly generate a report of what you have set up
• sg setup -fix to fix all issues with your dev setup in one go
• sg lint -fix to automatically try and fix your lint issues (only supported by a few linters at the moment, but more can easily be added!)
• Continuous integration testing for sg setup

dev/schemadoc has been removed and is replaced by sg migration ... commands. Example: sg migration describe -db codeintel --format=psql -force -out internal/database/schema.codeintel.md to generate the schema for the codeintel db thanks to #35905.

./dev/generate.sh has been deprecated in favour of of sg generate go (and its alias sg gen go for teammates in a hurry).

sg generate go now displays a progress bar to indicate its status and is also noticeably faster thanks to 35807, #36742 and #36681.

Named migrations files: sg migration now supports migrations with migration names embedded in the migration’s directory, and all newly created migrations from sg migration add will now have migration names included, which will make the migrations directory a bit easier to browser. #37244

Readability improvements: sg start logs are now easier to read, as the command names text is now justified. To ensure it’s still readable in small terminals, a few of them have be shortened and the enterprise- prefix is now implicit whereas oss- prefix has been introduced.

sg analytics: The DevX team is experimenting with collecting analytics on sg usage and issues! For now this is a manual process, so if you’d like to contribute you data to our explorations, you can submit your data with sg analytics submit [username], or check out what data has been collected with sg analytics view!

ADRs ❤️ sg: You will soon be able to list, search, view, and create Architecture Decision Records directly from sg! #37718

CI improvements

• The linter job that runs on every build is now inferring which linter task needs to run depending on the changes (except main where it runs everything), saving some time on pull requests thanks to #35331.
• When you retry the sg lint step, the verbose flag will be added allowing you to see more of what is going on.
• You can now force the run of tests that are executed when your PR is ready for review by specifying in your commit message [ready-for-review]. Gone are the days of flipping your PR between draft and ready for review!

Tech Radar

You can browse this month tech-radar to get a bird’s-eye view of Sourcegraph tech stack! Check this guide to learn about how updating it with your team initiatives.

DevOps and DevX team merge

As you may have heard, the Cloud DevOps team is splitting up into a new Cloud team, with the remaining teammates merging into the DevX team - so we’re happy to welcome 3 new teammates to the DevX team!

A consequence of this change will be a shift in ownership of various domains - this is still a work in progress, but you can see an overview in the Cloud DevOps handoff plan. Broadly speaking, the main changes are that the DevX team will soon own and lead initatives on the following fronts:

• observability (including internal tooling and external services like Grafana Cloud, Sentry, etc)
• the operation of sourcegraph.com
• misc. ops-related support

We will have more to share soon as the dust settles!

Logs, logs, logs 🗃️

A brand new logging package is now available in github.com/sourcegraph/log. This library outputs a OpenTelemetry-compliant log format in JSON mode, paving the way towards enabling customers to more easily ingest and leverage logs, and also offers a performant, strongly-typed interface for providing log fields. The library also encodes a number of best practices:

1. No global loggers - it is no longer possible to instantiate a logger at compile time, and users should hold their own references to loggers, so that fields can be attached and log output can be more useful with additional context, and pass Logger instances to components as required.
2. Loggers are attached to scopes - this helps orient log entries within a larger codebase. For example, when creating a GitHub V3 client for making a auth provider to make requests, one might use log.Scoped("provider.github.v3", "provider github client").

This library will also be the target of many observability improvements going forward, such as automated error reporting, improved test output, and more. To learn more, check out the new How to add logging guide.

We’ve also extended the existing internal/observation package, which aims to provide all-in-one logging, tracing, and monitoring primitives, to integrate logging throughout all levels of an observation. This enables you to easily write logs that includes helpful context like traces, metadata, observation context, and more. To learn more, check out How to add observability.

The DevX team has been collaborating with teams to help migrations to the new logging library - we encourage everyone to start incrementally migrating their existing logging, and to reach out to #dev-experience for feedback and questions!

sg experience ✨

The sg experience has been a major focus for the DevX team and we have been working towards a variety of improvements for both users and contributers of sg!

First up, usage improvements:

• sg now supports autocompletions that you can trigger using the Tab key to help you type out long command names and flags faster, and to help you learn sg commands faster! To get started, make sure you’ve run sg setup. (#33817)
• Many sg flags now have short aliases (such as sg run -d enterprise-frontend),and commands can declare shorter aliases too (such as sg ci st) to save you on some extra keystrokes.
• Misspelled a command? sg will now prompt you with some suggestions! (#33943)
• Help text is much improved, with sg help now rendering commands by category.
• sg lint has seen a variety of improvements, and now powers all linters that we run in CI, which means you can easily replicate linter runs locally for debugging and enabling developers to customize linter output with much more granularity.
• sg’s autoupdate mechanism has gone through a number of iterations and should now be reliably auto-updating your sg installation seamlessly whenever you run sg.

For developers wanting to streamline their developer experience with sg functionality, we’ve also made a lot of internal improvements:

• Linters are easier than ever to build with the updated lint.Runner interface, which now also provides you an easy way to get changed files and iterate over added lines to perform incremental linting. To get started, just head on over to the dev/sg/linters package!
• The migration to a new CLI library, urfave/cli, includes features like:
  • A much nicer API for defining flags and fetching them without declaring global variables, and convenience functions for safely getting arguments: example.
  • Developers can implement custom completions for their commands with the BashComplete: completeOptions(...) API.
  • Flags and commands can now have short aliases!
• sg.config.yaml can now leverage external secrets (we currently support gcloud only) with the new secrets: field, and sg commands can use the secretsStore.GetExternal(...) API.
  • This is used by sg test frontend-e2e, which you can use to run Sourcegraph’s E2E tests with ease! (#34627)
• The output API has been overhauled to be centralized in std.Out, which now centralizes the exports of a variety of sg-specific utilities for incorporating ✨ fancy ✨ output for some added bling. (#35269)
• Writing scripts? We strongly recommend everyone to start writing scripts in Go within sg, which gives us more code-sharing opportunities, better cross-platform compatibility, and more advanced features such as better output management. To enable this, the DevX team has started developing a new command execution library, github.com/sourcegraph/run, aimed at providing a seamless way to execute commands and manipulate its output in Go. (#35417)

Following your code from PR to production 🚢

Deployments are now announced over Slack, in #alerts-preprod-cloud for the preprod and in #deployments-cloud for Cloud deployments. If you want to receive a mention on those announcement when your PR is getting deployed, you can use the notify-on-deploy label. If the label is present when the PR is deployed you’ll receive the notification.

Deployements schedules can be observed in Honeycomb Dashboard which tracks how much time elapsed from the moment a PR being merged to the moment it got deployed.

Smoke testing ☁️

Every ten minutes, smoke tests are being run against Sourcegraph CLoud, first making basic infrastructure tests then performing a quick search to ensure that the application is up. And it’s for real this time, #16589 turns failures into an automated incident being created.

Learning resources 🎥

Check out this selection of some of our recently published learning resources!

• The Sourcegraph Codebase Tour video gives an overview of our main repository and what’s in it.
• The Tour of Secondary Repositories goes over some of the secondary repositories at Sourcegraph.
• Local development with sg setup is a video demonstrating how to use sg setup to set up your local development environment.

Architecture decision records 📰

The idea behind architecture decision records, or ADRs, is to have small documents that are part of the codebase, not an external artifact that you have to be aware of like an RFC.

We encourage the use of Architecture Decision Records (ADRs) for logging decisions that have notable architectural impact on our codebase. Since we’re a high-agency company, we encourage any contributor to commit an ADR if they’ve made an architecturally significant decision.

Note that ADRs are not meant to replace our current RFC process but to complement it by capturing decisions made in RFCs. However, ADRs do not need to come out of RFCs only. GitHub issues or pull requests, PoCs, team-wide discussions, and similar processes may result in an ADR as well, allowing to keep everyone in touch.

To learn more, check out the ADR index page and ADR 1: Record architecture decisions.

Database migration tooling 🗃️

There are two new goodies for database tooling available via sg migration locally and via the migrator binary shipped with every Sourcegraph instance.

• New describe command provides a formatted version of your database’s current schema
• New drift command provides a diff of the expected schema and your database’s current schema

And for some added bling, both of the new commands have been beautified! (#35722, #35735)

Preproduction environment 🔬

Before going out into production on Cloud, all changes are going throuh the preprod environment. The preprod environment is running in DotCom mode with a smaller dataset but with similar resources. Notably, it’s running some services which are sharded in production, but not within CI, at the miminum size that enables to exercise all code paths. This opened the path to increase our confidence toward changes through automated testing that weren’t previously possible.

Because tests on the prepod requires to put the application in a specific state to perform testing, state is being restored based on a snapshot which makes deterministic (#16249)

As a result, #16301 the code intel QA test suite is now running in preprod, and others will follow shortly.

Buildkite foundations ⛵

Since the end of March, 100% of our CI builds are now run on stateless agents. It means that by design, it’s now guaranteed that a given build cannot impact following ones. This enabled to roll out our own autoscaler backed by an in-house buildkite-job-dispatcher. This resulted in a whopping 30% decrease of our CI spending on GCP in April!

To try and maintain parity with the stateful agents of old, we have implemented a variety of measures to keep CI times down:

• Cross-node git repository mirrors means that repository cloning is consistently just as fast - if not faster! - than stateful builds.
• asdf caching has been used to speed up the installation process of all languages and tools needed to run our CI builds as we have now been running all builds on stateless agents. It has been extracted into a plugin, making it available for other pipelines as well.
• We have enabled image streaming for our CI cluster, which has reduced the time to pull an image and start it from 1m50s to ~2s, which means lower wait times for your CI builds. (infrastructure#3296)

Some other CI goodies include:

• Many linting steps have been rolled into a new CI step powered by sg lint, which now generates output that is actually readable compared to the “Misc linters” step of old! If you run into issues, (hopefully) helpful annotations will also be added to your build summary.
• Sending out a Slack mention when a specific step failed is also available as a plugin, which is also useful to make sure to notice a failure on a single step independently of the build result. For example, this is being used to alert the Code Intel team if their test suite fails on a preprod build.
• Changes to the client app now render app previews running against k8s.sgdev.org! Learn more in the Sourcegraph docs: Exploring client changes with PR previews

Tech Radar 💡

Thoughtworks Technology Radar is a well known source for getting a sense of where the technology landscape is going. What really makes it stand out is how it surfaces Thoughtworks opinions in a format that is really easy to process. What if we had if we used the same medium to keep everyone in the loop within Sourcegraph on our various initiatives on the engineering front?

Thanks to #35538, it’s an ongoing exploration that could potentially help all of us to stay in touch with all the ongoing initiatives at Sourcegraph in the blink of an eye. Feedback and ideas are welcomed on the PR!

SOC2 compliance processes

A new bot, pr-auditor, is now live in sourcegraph/sourcegraph and is rolling out to a number of other repositories that houses code that reaches customers. pr-auditor will add status checks on your pull requests when you edit descriptions to indicate whether or not it has detected a “test plan” within your pull request description. If a “test plan” is not provided by the time a PR is merged, an issue will be created in the sec-pr-audit-trail repository requesting that the PR author document a test plan, or provide a reason for the exception. This serves as an audit log to help us achieve these two SOC2 control points:

GN-104 Code changes are systematically required to be peer-reviewed and approved prior to merging code into the main branch.

var err errors.MultiError
for _, fn := range thingsToDo {
  err = errors.Append(err, fn())
}
return err

import "github.com/sourcegraph/sourcegraph/dev/internal/team"

func main() {
  // Neither a GitHub client nor a Slack client is required, but each enables more ways
  // to query for users and/or get additional metadata about a user.
  teammates := team.NewTeammateResolver(githubClient, slackClient)
  tm, _ := teammates.ResolveByName(ctx, "Robert")
  println(tm.SlackID)
  println(tm.HandbookLink)
  println(tm.Role)
  // etc.
}

sg ci build main-dry-run