Should Every Company Build a Big Data Team? Insights and Expert Opinions

Background

Since the publication of Victor Shuenberg’s 2012 book The Big Data Era and the Chinese State Council’s 2015 Action Plan for Promoting Big Data Development, “big data” has become a widely used term. Many enterprises claim to operate big‑data projects, but the necessity of a dedicated big‑data department is still debated.

Actual Market Demand

Research from SAP shows that 95 % of companies regularly process only 0.5 TB – 40 TB of data . Only the largest technology firms routinely handle petabyte‑scale workloads. A separate study of U.S. companies indicates that more than 50 000 firms with 20–500 employees also face data‑processing challenges, suggesting that the primary market for big‑data capabilities lies in the “Fortune 500 000” segment rather than the top‑50.

Practical Definition of Big Data

Instead of the classic 3 V definition (volume, velocity, variety), big data can be treated as a subjective state : a situation where an organization’s current infrastructure cannot meet its data‑processing requirements. The definition therefore focuses on a capability gap rather than an absolute data size.

Common Pitfalls

Many organizations advertise terabytes or petabytes of stored data and large Hadoop or Kafka clusters, yet they suffer from low‑quality data. Experian Data Quality reports that 88 % of enterprises experience financial impact from inaccurate data , with revenue losses up to 12 % . The “garbage‑in, garbage‑out” problem undermines any potential competitive advantage.

Evaluating the Need for a Dedicated Big‑Data Team

When deciding whether to create an internal big‑data department, consider the following criteria:

Data Volume vs. Business Value : Estimate the daily ingest rate, total storage, and the expected ROI of analytics projects. If the projected value does not outweigh the cost of hiring engineers, building pipelines, and maintaining clusters, an internal team may not be justified.

Data Quality and Governance : Assess the current data cleansing, metadata management, and lineage processes. Poor data quality will negate the benefits of scaling infrastructure.

Technical Expertise : Determine whether existing staff possess expertise in distributed storage (e.g., HDFS, S3), stream processing (Kafka, Flink), and large‑scale analytics (Spark, Hive, Presto). If not, the learning curve and recruitment costs can be prohibitive.

Infrastructure Costs : Calculate total cost of ownership (hardware, cloud instances, networking, licensing, and operational overhead). Compare this with subscription fees for managed services such as Amazon EMR, Google Dataproc, or Azure Synapse.

Regulatory and Security Requirements : Some industries (finance, healthcare) may require on‑premises control of data, influencing the decision toward an internal platform.

Alternative Approaches

For most small‑ and medium‑size enterprises (SMEs), the following options are more cost‑effective than building a platform from scratch:

Leverage Open‑Source Stacks : Deploy community‑supported components (Hadoop, Spark, Kafka, Airflow) on cloud VMs or container orchestration platforms (Kubernetes). This reduces licensing fees while retaining flexibility.

Adopt Managed Cloud Services : Use fully managed offerings (e.g., AWS Glue, Google BigQuery, Azure Data Lake) to offload cluster provisioning, scaling, and maintenance.

Outsource to Specialized Vendors : Contract third‑party providers that deliver end‑to‑end pipelines, data lakes, or analytics‑as‑a‑service. This allows the organization to focus on core business logic while the vendor handles data engineering.

Expert Perspectives

Wei Xinghua (Woqi Technology) recommends a cost‑benefit analysis and suggests purchasing third‑party big‑data services when the internal ROI is insufficient.

Chen Songzheng (Hunan Qilin Information Engineering) distinguishes between data suppliers (who need dedicated teams) and data consumers (who can rely on external platforms). He predicts that data analysts will become essential roles for all consumers.

Li Xiupeng (Sohu Video Recommendation System) points out that a massive user base and a need for real‑time recommendation algorithms justify a dedicated department, whereas companies without such scale may not need one.

Decision Framework

To decide whether to establish a big‑data department, follow this workflow:

1. Quantify data volume and growth rate.
2. Identify critical analytics use‑cases and estimate their business impact.
3. Evaluate existing data quality and governance processes.
4. Compare total cost of ownership (internal team + infrastructure) vs. managed service fees.
5. Consider regulatory constraints and security requirements.
6. Choose one of:
   a) Build internal platform (if ROI > cost & compliance demands on‑prem).
   b) Use managed cloud services (if scalability & speed are priorities).
   c) Outsource to a specialist vendor (if internal expertise is lacking).

Conclusion

For most small companies, constructing a big‑data platform from the ground up is rarely cost‑effective. Leveraging open‑source components, managed cloud services, or specialized vendors provides the necessary capabilities without the overhead of a dedicated internal department. The ecosystem of big‑data service providers continues to mature, offering scalable solutions that align with the actual data‑processing needs of the majority of enterprises.