Reflections on DevOps Transformation, Metrics, and UI Automation Testing after a Visit to Microsoft

In recent years, terms like Scrum and DevOps have become ubiquitous in the software industry, yet there is no single definition of DevOps, leading to varied interpretations and no standard answer for implementing it effectively.

During a brief exchange with Microsoft’s Seattle headquarters, the author observed similarities between Microsoft’s DevOps transformation practices and their own team’s operations, emphasizing the value of learning from other organizations.

The discussion revealed a set of metrics often used in DevOps contexts, including Usage, Acquisition, Engagement, Satisfaction, Churn, Feature Usage, Velocity, Time to Build, Time to Self Test, Time to Deploy, Time to Learn, Live Site Health, Time to Detect, Time to Communicate, Time to Mitigate, Customer Impact, Incident Prevention Items, Aging Live Problem, SLA Per Customer, Customer Support, Original Estimate, Completed Hours, Lines of Code, Team Capacity, Team Burndown, Team Velocity, and Bugs Found.

The author notes that focusing on delivery speed within the DevOps loop means measuring how quickly a story can be released after completion, rather than estimating story size beforehand, as rapid delivery often outweighs precise estimation.

When asked about UI automation testing, the author learned that while UI tests can be technically straightforward, they often require substantial maintenance due to UI changes, making full automation challenging; a balanced approach with integration testing, unit testing, and API testing is preferred.

Regarding Microsoft’s DevOps transformation, the author observed that Microsoft reduced its QA team, shifting quality responsibilities to developers and product managers, which can streamline automation but also requires careful bug prioritization to avoid delaying releases.

Overall, the experience highlighted that while the author’s team shares many DevOps practices with Microsoft, industry-specific differences mean that direct copying is not feasible, and adaptations must align with the organization’s culture and processes.