Understanding Hive Execution Engines: MapReduce, Tez, and Spark – Principles, Optimization, and Explain Usage

The article begins with a brief history of Hive, noting its evolution from a slow MapReduce‑based system to a flexible platform that now supports multiple execution engines such as MapReduce, Tez, and Spark.

MapReduce Engine : It describes Hive's five core components (UI, DRIVER, COMPILER, METASTORE, EXECUTION ENGINE) and outlines the six compilation stages—lexical/semantic parsing, logical plan generation, logical optimization, physical plan generation, and physical optimization—illustrated with sample SQL and corresponding abstract syntax trees.

Example code for the first compilation stage is shown as

词法、语法解析: Antlr 定义 SQL 的语法规则，完成 SQL 词法，语法解析，将 SQL 转化为抽象语法树 AST Tree

, and a full SELECT statement is presented in select * from dim.dim_region where dt = '2021-05-23';.

Explain Syntax : The article explains Hive's EXPLAIN [EXTENDED|CBO|AST|...] query command, its output sections (stage dependencies and stage plans), and demonstrates how to interpret operator trees, predicates, and join behavior.

Tez Engine : It introduces Tez as a DAG‑based framework that splits Map and Reduce tasks into finer‑grained components, lists its six programmable interfaces (Input, Output, Partitioner, Processor, Task, Master), and compares Tez to traditional MapReduce, highlighting reduced I/O and faster execution. Configuration examples such as

<property><br/>    <name>hive.execution.engine</name><br/>    <value>tez</value><br/></property>

are provided, along with memory and container tuning parameters.

Spark Engine : The article covers Hive on Spark, describing how Hive tables are converted to RDDs, the role of SparkCompiler in building MapWork and ReduceWork, and the execution flow using foreachAsync. It details Spark‑specific settings (executor memory, cores, dynamic allocation) and Hive‑Spark tuning flags such as hive.auto.convert.join.noconditionaltask.size and hive.stats.collect.rawdatasize.

Throughout, practical SQL examples (joins, group‑by, sub‑queries) and their corresponding Explain outputs illustrate how Hive optimizes queries across different engines, and the article concludes with best‑practice recommendations for performance tuning and container pre‑warming.