AI-Era Data Engineering: Emerging Tech Stacks & Career Paths from Job Market Analysis

The article opens with a typical job description for an AI-era data development role, highlighting requirements beyond traditional stacks like Spark, Flink, and Kafka. New requirements include large model post-training, RAG, Agent, multimodal, vector retrieval, evaluation systems, and AI data lakes. Candidates are not expected to master all areas but should have depth in at least one.

These positions tend to be platform-oriented yet deeply tied to business, concentrated in large companies and AI startups with high entry barriers for education and experience. A reference to a Bilibili video provides additional visual overview: https://www.bilibili.com/video/BV1wNb16BEng.

The author characterizes Agent development as low-barrier, falling within backend development scope, and suggests that engineers with solid backend foundations can learn Agent basics in about a week.

Part-1: Re-establishing Cognition

Defines emerging roles: AI Data Engineer, AI Data Pipeline, RAG, Agent, post-training. Raises questions about each direction's responsibilities and future career ceilings.

Part-2: RAG

Enterprise-grade RAG involves multiple stages following the workflow:

Document
Chunk
Embedding
Vector
BM25
Hybrid Search
Reranker
Context
Eval
...

The author emphasizes RAG is not exclusive to data engineering and involves broader enterprise considerations.

Part-3: Post-training

Covers data cleaning, deduplication, supervised fine-tuning (SFT), and data generation.

Part-4: Agent Engineering

Describes core Agent components: MCP, SKILL, Memory, Tracing, Eval.

Part-5: AI Infra

Includes multimodal processing, Ray, and Lance.

The author concludes that mastering 1-2 of these areas is sufficient depending on employer needs, and future content will explore each module in depth.