DSM Release Process: Weekly and Feature Deployments, Challenges and Solutions

Only when a product is successfully released to users and receives positive feedback does the team's value become evident.

The author asks readers to indicate their project's release cycle from five options (no fixed cycle, weekly, bi‑weekly, monthly, longer).

From 2013 to the present, the author has overseen three major milestones for Google DoubleClick Sales Manager (DSM): Beta, GA, and integration with Ad Exchange.

1. Regular Weekly Release

The release criterion is simple: no P0/P1 bugs.

The release flow is illustrated below.

Automatically fetch the latest three‑hour green CL from TAP (Test Automation Platform) and build a new version.

Deploy to the QA environment.

Run automated and manual tests in QA.

If P0/P1 bugs are found, hand them to developers for fixes, then rebuild and redeploy to QA.

After QA sign‑off, promote to the Canary environment.

Canary runs for 3–6 hours while monitoring the eye3 system.

If eye3 reports no errors, the release proceeds to production.

Rapid (Release Automation Platform) creates a candidate package, runs webdriver tests, and publishes to QA, automatically generating release bugs and hotlists.

After QA sign‑off, the package is rolled out to Canary and then Production.

2. Feature Release

Each new feature has an independent feature flag.

A dedicated bug is created for each feature to provide a unified communication channel.

Feature launch forms contain several checklist items (multiple screenshots shown).

If a serious issue appears after launch, the feature can be quickly disabled.

3. Common Issues and Solutions

During the mandatory process, exceptions arise:

TAP cannot retrieve a green CL within three hours:

Form a three‑person build‑cop team to monitor TAP greenness on Fridays and Mondays.

If TAP remains red for over 10 hours, trigger emergency handling to ensure a reliable baseline CL for weekly releases.

P2 bugs discovered in regression may actually be P1:

Technical Leaders re‑review such bugs to decide if a cherry‑pick is required.

Require a sign‑off after regression testing.

How to know when a new feature is ready for testing:

Obtain the last CL from the feature bug and check its status via the cl‑status tool (screenshot shown).

Use an auxiliary Chrome extension to assist.

How developers can quickly obtain error logs for fast bug fixing:

Leverage the same Chrome extension for log collection.

Post‑release incident handling:

Eye3 monitoring, on‑call rotation, rollback procedures, cherry‑pick, and post‑mortem (autopsy) analysis.

4. Benefits of This Process

QA team’s weekly time allocation becomes predictable.

Weekly release statistics help assess product stability.

Reduced regression time and fewer cherry‑picks.

More time is available for new‑feature testing and automation.

5. Future Improvements

API‑centric accelerated releases: decouple API from FE, increase test coverage, move from weekly to Thursday releases.

Compatibility testing for multiple API versions after decoupling.

Schedule releases on Thursdays to allow issue resolution on Fridays without weekend overtime.

Personal reflection: after several years of weekly releases, the pressure on the team is high; while regression time dropped from two days to one, more than three new features per week leave no breathing room. A bi‑weekly cadence is considered more ideal.

Author: Guo Wen, Google SWE on Testing.