From 1.0 to 3.0: Evolution of a Data Sandbox Platform on AWS EMR

Data Sandbox 1.0

Implemented as a single‑master EMR cluster with S3 (pupumall‑sandbox bucket) as the primary store. Services included Zookeeper, HDFS, YARN, Spark, Hive, Zeppelin, JupyterHub, Livy and Hue. Access control relied on AWS IAM Roles. In a scenario of 100 users accessing 500 tables, the number of IAM policies would exceed a thousand, creating high maintenance overhead. Additional drawbacks were lack of data isolation, inefficient sync/lifecycle management, no unified masking, and unrestricted user access to all sandbox data, leading to the decision not to launch version 1.0.

Data Sandbox 2.0

Switched storage to HDFS and introduced Apache Ranger for fine‑grained, user‑level permissions. Architecture added a data‑sync pipeline that recorded metadata in MySQL and Redis, automated masking, and lifecycle management. Key components: HDFS main store, Spark jobs submitted via Zeppelin, Ranger‑mediated authorization, optional S3 access for low‑sensitivity data.

Problems observed after six months of operation:

Storage cost: SSD gp2 on EBS cost ≈ 3× S3 (¥0.746 / GB / month). For 9 TB with 2× replication and 20 % redundancy, annual cost ≈ ¥198 k; projected three‑fold growth would reach ≈ ¥600 k / year.

Scalability limited; node‑failure impacted service availability.

Sync logic complex; daily resource consumption ≈ 500 CPU cores and 1000 GB RAM.

Data availability delayed ~6 h after production ETL.

Permission mapping cumbersome because policies were path‑based rather than object‑based.

Data consistency issues from re‑sync and cleaning processes.

Data Sandbox 3.0

Core changes: retain S3 as the primary data lake, keep HDFS only for logs and user results, enforce access control at the Spark‑SQL layer using Ranger‑SparkSecurity (derived from Apache Submarine). Security components added: Kerberos for authentication, Ranger for authorization, SSL/TLS between components, AWS Secrets Manager for certificate handling, and a RecordServer for fine‑grained request auditing.

Implementation steps:

Create a dedicated IAM Role with read‑only access to the production S3 bucket and read/write access to the sandbox bucket; attach the role to all EMR nodes.

Adapt spark‑security to support INSERT OVERWRITE operations.

Patch createTempView to prevent unauthorized path access.

Identify masking bugs (UDF usage and UNION‑based queries) and mitigate by upgrading spark‑security and adding Ranger‑level column policies.

S3 Data‑Access Control Implementation

IAM Role creation: grant EMR cluster read‑only permission on the production bucket and read/write permission on the sandbox bucket; all EMR components reference this role.

Spark‑security adaptation: detach from Apache Submarine, rebuild for Spark 3.1 and Ranger 2.1, add INSERT support, and modify createTempView execution plan to block paths pointing to the production bucket.

Masking issue resolution:

UDF scenario: upgrade spark‑security to the latest version where the issue is fixed.

UNION scenario: apply column‑level Ranger policies; data‑masking on subsequent UNION branches is not supported without extensive code changes.

Data Management Optimizations

Eliminated double‑sync by directly syncing Hive table schemas across clusters. Developed a custom Hive‑metadata cross‑cluster sync tool to keep production and sandbox schemas aligned.

Introduced configurable data‑masking policies within Ranger and refined lifecycle management to operate on table schemas with automated retention rules (daily, weekly, monthly, full).

Benefits

Compute cost reduction up to 40 %.

Storage cost reduction 90 % (≈ ¥60 w / year).

Query performance improvements: adaptive stage sizing, skew‑join optimization.

Near‑real‑time data availability; query window extended beyond two weeks.

Administrative gains: fine‑grained object‑level permissions (library‑table‑field), group/user policies, centralized masking, simplified sync configuration, lower operational complexity.

Overall, version 3.0 achieves a practical balance between security and convenience, delivering significant cost savings while improving data accessibility for business analysts.