Google App Engine Datastore: Usage, Architecture, and Implementation

Datastore is based on the concept of an Entity that resembles an object and can contain multiple Properties such as integers, floats, or strings. Because it is schema‑less, the application defines the schema and can easily add, remove, or modify properties. An Entity instance is analogous to a row, and a collection of entities of the same kind forms a Kind . Datastore is designed for hierarchical data with Root Entities and Child Entities, which together form an Entity Group that can be stored in a single BigTable partition and support local transactions.

Advanced features include the Google Query Language (GQL), a small subset of SQL supporting operators like >, <, =; automatic index generation by App Engine (except for composite indexes); distributed design that typically returns results within 200 ms; and support for relationships via ReferenceProperty in the Python API.

The following table compares Datastore with traditional relational databases:

Datastore

Relational Database

SQL Support

Only basic queries

Full support

Main Structure

Hierarchical

Relational

Index

Partially auto‑created

Manual creation

Transaction

Only within an Entity Group

Supported

Average execution speed (ms)

<200

<100

Scalability

Very good

Difficult, requires extensive changes

In terms of APIs, the Python version provides a private API that is easy to learn for basic operations but harder for advanced features like relationships and transactions, while the Java API follows JDO/JPA standards with some differences from Hibernate.

Implementation Details

Datastore is built on top of Google’s BigTable . BigTable is essentially a massive table that stores rows, each with a name and a set of columns. To handle massive data, BigTable shards the table across many servers and sorts it for efficient queries.

BigTable supports basic CRUD operations, single‑row transactions, and prefix/range scans.

The Entities Table stores all entities in a single column per row, using a serialized form of the entity. The entity key is derived from its parent hierarchy, e.g., "/Grandparent:Ethel/Parent:Jane/Child:Timmy".

Indexes are stored in separate BigTable tables to accelerate queries. Three main index types exist:

Kind Index : automatically generated to retrieve all entities of a given kind.

Single‑property Index : automatically generated for each property value, with separate tables for ascending and descending order.

Composite Index : manually defined by developers for queries involving multiple properties.

Transactions are performed using BigTable’s single‑row transaction mechanism combined with Optimistic Concurrency Control. Writes read the entity’s committed timestamp, log the write serially, and update the timestamp if no conflict occurs; otherwise the transaction is retried. Multi‑entity transactions require the entities to belong to the same Entity Group, ensuring they reside on the same physical machine.

Backup for Datastore operates at the Entity Group level and uses the Paxos algorithm, offering stronger safety guarantees than BigTable’s row‑level backups.

Overall, Datastore’s design differs significantly from relational databases: while it may not match relational databases in raw write speed, its hierarchical model, automatic scaling, and read‑optimized performance make it well‑suited for modern web applications that require massive, easily scalable data storage.