How Atlas’s Metadata Storage Model Enables Scalable Hadoop Governance

Atlas is an extensible core governance service suite that helps enterprises meet compliance requirements in Hadoop and integrate with the broader data ecosystem. It manages shared metadata, data classification, auditing, security, and data protection.

Overall Architecture

Atlas consists of four major parts: Core (type system, import/export, graph engine), Integration (REST API and Kafka I/O), Metadata Source (plugins for capturing metadata from big‑data services), and Apps (web UI, tag‑based policies, business classification systems).

Type System

Atlas allows users to define a model for the metadata objects they wish to manage. The model is expressed as "types"; instances of these types are called "entities". Types are divided into native types (Int, String, Boolean, Date, Enum, etc.) and structured types (Array, Map, Class, Struct, Trait). Structured types serve as the building blocks for business‑specific metadata models.

Two main categories of structured types are used:

Native types: basic primitives and enums.

Structured types: collections (Array, Map) and composite types (Class, Struct, Trait).

Metadata entities inherit from these type definitions. For example, a Hive table entity is modeled using the predefined DataSet type, while computational metadata uses the Process type. The storage schema of a type must satisfy the same constraints as a relational table schema.

Graph Storage Structure

All Atlas data—type schemas and concrete entities—are stored as a graph implemented by JanusGraph, which in Tencent’s deployment uses HBase as the underlying storage engine. The graph consists of three elements: vertex, edge, and property.

JanusGraph adopts a point‑split storage model: each vertex is stored as a single HBase row, with its properties and all incident edges placed in columns of that row.

Mapping Types to JanusGraph

Each Atlas type definition is translated into a JanusGraph vertex with a set of properties and, when necessary, outgoing edges. The following code snippets illustrate the vertex schema for the five major type categories.

vertex属性 __type="typeSystem" __type.category=TypeCategory.ENUM __type.name=enumName ... __type.enumValueName=ordinal

#vertex属性
__type="typeSystem"
__type.category=TypeCategory.STRUCT
__type.name=structName
... __type.${typeName}.${attrName}=json(AtlasAttribute)
... __type.edge.${typeDefName}.${attriDefName}

#vertex属性
__type="typeSystem"
__type.category=TypeCategory.CLASSIFICATION
__type.name=classificationName
... __type.supertype   # edge to super type
... __type.entitytype  # edge to allowed entity types

#vertex属性
__type="typeSystem"
__type.category=TypeCategory.ENTITY
__type.name=entityName
... __type.edge.${typeDefName}.${attriDefName}
... __type.supertype   # edge to super type

#vertex属性
__type="typeSystem"
__type.category=TypeCategory.RELATIONSHIP
__type.name=relationshipName
... endDef1=json(AtlasRelationshipEndDef)
... relationshipLabel
... edgeLabel determined by end definitions

Metadata Entity Graph Model

An entity such as a Hive table is first represented as JSON, then converted to graph vertices. Primitive attributes become vertex properties; struct attributes become edges to their own struct vertices; map and array attributes are handled according to the value type (primitive, enum, struct, reference). Classification information is stored by adding the classification name to the __traitNames property, creating a classification vertex, and linking it with a classifiedAs edge.

{
  "typeName":"tencent_bg",
  "attributes":{
    "owner":"ls",
    "qualifiedName":"CSIG@tencent",
    "name":"CSIG",
    "description":"csig business group"
  },
  "guid":"-1234567890",
  "proxy":false,
  "version":0
}

Type Parsing and Validation

Atlas validates every type definition in three phases:

Phase 1 : Checks native‑type constraints, reference existence, collects all attributes (including unique ones), and resolves super‑type hierarchies for structs, entities, classifications, and relationships.

Phase 2 : Resolves inverse references, populates sub‑type collections for all parent types, and determines relationship edge labels and directions based on isLegacyAttribute flags.

Phase 3 : Emits warnings for missing relationships, merges super‑type relationship attributes, and validates classification‑entity compatibility, throwing exceptions when constraints are violated.

Conclusion

The article walks through Atlas’s storage architecture, type system, graph‑based persistence, and the rigorous parsing process that together enable a robust, extensible metadata governance platform for Hadoop ecosystems.