Overview of Hive Data Warehouse, Its Architecture, Query Processing, and Comparison with Impala

Hive is a Hadoop‑based data warehouse that offers a SQL‑like language (HiveQL) to query data stored in HDFS, converting queries into MapReduce jobs for batch processing.

The article first explains data‑warehouse concepts and architecture layers—data source, integration (ETL), storage/management, services, and applications—and highlights the limitations of traditional relational warehouses for massive, diverse data.

It then describes Hive’s components: the user‑interface module (CLI, HWI, JDBC, Thrift Server), the driver module (compiler, optimizer, executor), and the Metastore (metadata repository), and shows how Hive interacts with other Hadoop ecosystem tools such as Pig, HBase, and Mahout.

Detailed examples illustrate how HiveQL statements are turned into MapReduce tasks. For a join operation, Hive generates key‑value pairs in the map phase, shuffles them by key, and performs the join in the reduce phase. Example code:

SELECT name, orderid FROM User u JOIN Order o ON u.uid=o.uid;

For a group‑by operation, Hive maps each record to a composite key <rank, level>, shuffles by that key, and aggregates counts in the reduce phase. Example code:

SELECT rank, level, count(*) as value FROM score GROUP BY rank, level;

The Hive query workflow is outlined: Antlr parses the SQL, builds an abstract syntax tree, creates a QueryBlock, generates an OperatorTree, applies logical and physical optimizations, and finally launches one or more MR jobs via an XML execution plan.

Hive high‑availability (HA) architecture is introduced, where multiple Hive instances are pooled behind HAProxy; the proxy performs health checks, routes client requests to healthy instances, and manages failover and instance restarts.

Impala, a low‑latency SQL engine from Cloudera, is presented as an alternative to Hive’s batch processing. Impala accesses HDFS/HBase directly without converting SQL to MR, offering significantly lower query latency while reusing Hive’s Metastore.

Impala’s architecture consists of Impalad processes, a State Store for cluster metadata, and a CLI. Its query execution includes registration, planning, metadata lookup, task distribution, result aggregation, and returning results to the client.

A side‑by‑side comparison highlights that Hive excels at long‑running batch analytics, whereas Impala is optimized for interactive SQL queries; both share storage formats, metadata, and SQL syntax but differ in execution models and resource handling.