Optimizing TimesTen Cache Group Rebuilds During Oracle 19c Migration

Background

TimesTen (TT) is an Oracle in‑memory relational database that provides ultra‑low latency and high throughput. It integrates with Oracle DB via a native Cache Group mechanism that keeps selected Oracle tables synchronized with TT tables in real time.

Cache Group Types

Read‑Only : TT is read‑only; DML statements are passed through to Oracle.

Synchronous writethrough (SWT) : DML is executed in TT first, then committed to Oracle; errors cause rollback in TT.

Asynchronous writethrough (AWT) : DML is executed in TT and returns immediately; a Cache Agent applies changes to Oracle asynchronously.

The migration project primarily used the Read‑Only type to cache user‑information tables for a billing application.

Migration Scenario

Two regional schemas were moved from an IBM AIX mainframe running Oracle 11g (RAC, 2 nodes) to a domestic X86 appliance running Oracle 19c (RAC, 6 nodes). The environment also included two TT clusters (primary‑standby) linked to Oracle via Cache Groups.

Source Oracle

Host: IBM E880

OS: AIX 7.1

Version: 11.2.0.4

RAC: Yes (2 nodes)

Target Oracle

Host: domestic X86 appliance

OS: RedHat 7.6

Version: 19.5

RAC: Yes (6 nodes)

TimesTen hosts

IBM P780 (PowerPC)

OS: AIX 7.1

Version: 11.2.2.8.43

Recreation Steps

Stop all applications on the 11g database.

Stop GoldenGate replication and verify data consistency.

Delete existing Cache Groups on both TT clusters.

Recreate Cache Groups based on the new 19c database.

Steps 1‑3 completed without issue; step 4 stalled. Large tables loaded only ~33% of rows and the overall Cache Group rebuild time far exceeded expectations, causing time‑outs.

Root Cause Analysis

Oracle AWR reports showed significant Global Cache (GC) wait events during the rebuild, indicating heavy Cache Fusion traffic across the six RAC nodes. In the original 11g dual‑node RAC, Cache Group synchronization ran on a single node; after migration to a six‑node RAC the workload was distributed, increasing cross‑node memory accesses and GC contention.

Optimization Strategy

To reduce GC waits, the Cache Groups were partitioned by schema and bound to specific RAC nodes, ensuring that the data‑read workload for each group remained local to one node.

Create separate Cache Groups for the JY and SW schemas.

Assign each Cache Group to a dedicated RAC node (node‑affinity).

Recreate the Cache Groups following this node‑affinity design.

After applying the partitioning, rebuild time dropped to roughly one‑third of the original duration. Subsequent three‑region cut‑overs consistently achieved the same improvement.

Key Takeaways

Cache Group reconstruction is I/O‑intensive because it scans every cached table. In environments with multiple TT clusters and RAC nodes, an unpartitioned Cache Group can generate massive GC wait events and time‑outs. Planning Cache Groups per schema and anchoring them to specific RAC nodes isolates the read workload, dramatically speeds up rebuilds, and supports smooth migrations of large databases.