The Evolution of Data Science and Big Data at Microsoft

Original author: Mario Garzia, former partner and data‑science architect at Microsoft Research; translated by Du Hongguang.

Data science and “big data” have become buzzwords of the 21st‑century high‑tech industry. Historically, large‑scale data challenges existed long before the term, such as the 1880 U.S. Census, which took eight years to compile, and the 1890 Census, which was completed in under a year thanks to Herman Hollerith’s punched‑card system—the foundation of IBM.

Today, the challenges differ: data volume grows faster than ever, and the variety and velocity of data collection are also accelerating. Ericsson’s 2011 report predicted nearly 50 billion connected devices by 2020, each generating its own data, while the systems that manage this data create even more data.

Modern big data also presents huge opportunities. Companies can now collect data directly from end users to understand experience and service levels, enabling new products and unprecedented service quality. Tech giants have turned data into products (e.g., Bing search, social platforms), but the current focus is on democratizing data and analytics so that all industries can optimize services for their customers.

Microsoft has a long tradition of data‑driven decision making. Since I joined in 1997, I have witnessed the evolution from using data to understand products to using data to understand user experience and service. The culture of continuous learning and data‑centric thinking is deeply ingrained.

In 2000 I joined the Windows team and founded the Windows Reliability group. Reliability metrics were derived from about a century’s worth of server operation data. After releasing Windows Server 2000, we offered customers a free reliability service that collected and analyzed data‑center server data, providing insights that were previously unavailable to most enterprises. This data informed OS reliability improvements, failure‑mode analysis, and the development of new diagnostic services.

Today, Microsoft’s products and services focus not only on quality but also on deep user understanding. Data‑driven culture means every employee, not just data scientists, must be data‑savvy and use data to solve problems. Big data powers experiments, product improvements, and the deployment of machine‑learning models through Azure ML.

As a Microsoft data scientist, I have access to unprecedentedly broad user data—from PCs, tablets, phones, games, search, and many services—covering all aspects of users’ lives. This enables us to better understand needs, improve experiences, and create new, more effective ways to influence daily life. Data‑science principles sit at the core of Microsoft’s data‑driven strategy, with clear career paths for data scientists, machine‑learning scientists, and applied scientists, fostering a vibrant and growing community.