Spring Boot + Apache Ignite: Distributed Cache, SQL & Compute Grid Implementation Guide

When to Consider Ignite

Local caches like Caffeine offer nanosecond reads but cannot share data across JVMs and hit heap limits, causing Full GC pauses. Redis solves sharing but suffers from network round-trips (0.2–0.5 ms per GET), hot-key bottlenecks (single slot in Redis Cluster), and limited persistence guarantees (RDB/AOF for recovery, not strong consistency). Ignite combines distributed in-memory caching, ANSI SQL with secondary indexes, ACID transactions, and a compute grid that moves code to data — avoiding massive data transfers for analytics like real-time risk scoring over 90-day user behavior sequences. Native persistence unifies memory and disk, preventing cache-avalanche fallback storms. Trade-off: higher operational complexity.

Core Ignite Concepts

Memory Grid & Node Roles

Cluster nodes form a logical memory grid; clients see a single dataset. Server nodes hold data and partitions; client nodes ( setClientMode(true)) proxy requests without storing data. Business apps typically run as clients against a dedicated server cluster.

Persistence: Memory + Disk Unified

With persistence enabled, writes go to WAL first (crash durability), then background checkpoints flush dirty pages to data files. On restart, pages load on-demand. Memory becomes a cache for disk, allowing dataset larger than RAM. pageEvictionMode applies to pure in-memory regions, not persistent storage.

Partitioning & Replication

Each cache defaults to 1024 partitions via RendezvousAffinityFunction. backups=1 gives one primary + one backup on different nodes; failover is typically sub-second.

CacheConfiguration<Long, Order> cfg = new CacheConfiguration<>("orderCache")
    .setCacheMode(CacheMode.PARTITIONED)
    .setBackups(1)
    .setAffinity(new RendezvousAffinityFunction(false, 1024));

Affinity Collocation

Partition ownership is derived from affinityKey hash. If two tables share the same affinity key (e.g., orderId for Order and OrderItem) and identical affinityKeyMapper, their rows co-locate on the same node. JOINs then execute locally with zero network traffic. Without collocation, Ignite may broadcast one table to all nodes — expensive. Affinity keys must be designed upfront.

Cluster Discovery

Options: TcpDiscoveryMulticastIpFinder (dev only, multicast often blocked in cloud), TcpDiscoveryVmIpFinder (static IPs, ports 47500–47509), TcpDiscoveryZookeeperIpFinder, TcpDiscoveryKubernetesIpFinder (needs ignite-kubernetes module), TcpDiscoveryS3IpFinder / CloudIpFinder for auto-scaling.

Spring Boot Integration

Dependencies (Maven)

<properties>
  <ignite.version>2.16.0</ignite.version>
</properties>
<dependencies>
  <dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-core</artifactId>
    <version>${ignite.version}</version>
  </dependency>
  <dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-spring</artifactId>
    <version>${ignite.version}</version>
  </dependency>
  <!-- SQL indexing (H2 engine) -->
  <dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-indexing</artifactId>
    <version>${ignite.version}</version>
  </dependency>
  <!-- Spring Data Repository -->
  <dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-spring-data-3.0</artifactId>
    <version>${ignite.version}</version>
  </dependency>
</dependencies>

Java Configuration (Preferred over XML)

@Configuration
@EnableCaching
@EnableIgniteRepositories(basePackages = "com.demo.order.repo")
public class IgniteConfig {
    @Value("${ignite.client-mode:true}")
    private boolean clientMode;

    @Bean(destroyMethod = "close")
    public Ignite ignite() {
        IgniteConfiguration cfg = new IgniteConfiguration();
        cfg.setIgniteInstanceName("appGrid");
        cfg.setClientMode(clientMode);
        cfg.setPeerClassLoadingEnabled(false); // disable in prod
        cfg.setMetricsLogFrequency(0);
        cfg.setDataStorageConfiguration(dataStorageConfig());

        TcpDiscoverySpi spi = new TcpDiscoverySpi();
        if (isK8s()) {
            TcpDiscoveryKubernetesIpFinder finder = new TcpDiscoveryKubernetesIpFinder();
            finder.setNamespace("middleware");
            finder.setServiceName("ignite-service");
            spi.setIpFinder(finder);
        } else {
            TcpDiscoveryVmIpFinder finder = new TcpDiscoveryVmIpFinder();
            finder.setAddresses(List.of(
                "10.0.1.11:47500..47509",
                "10.0.1.12:47500..47509",
                "10.0.1.13:47500..47509"
            ));
            spi.setIpFinder(finder);
        }
        cfg.setDiscoverySpi(spi);
        cfg.setMarshaller(new BinaryMarshaller());
        return Ignition.start(cfg);
    }

    private DataStorageConfiguration dataStorageConfig() {
        DataRegionConfiguration region = new DataRegionConfiguration()
            .setName("default")
            .setInitialSize(512L * 1024 * 1024)   // 512 MB
            .setMaxSize(3L * 1024 * 1024 * 1024) // 3 GB off-heap
            .setPersistenceEnabled(true);

        DataStorageConfiguration ds = new DataStorageConfiguration();
        ds.setDefaultDataRegionConfiguration(region);
        ds.setStoragePath("/data/ignite/db");
        ds.setWalPath("/data/ignite/wal");
        ds.setWalArchivePath("/data/ignite/wal/archive");
        ds.setWalMode(WALMode.LOG_ONLY);
        ds.setCheckpointFrequency(180_000); // 3 min
        ds.setPageSize(4 * 1024);           // keep 4 KB for persistence
        return ds;
    }
}

Embedded server mode ( clientMode=false) simplifies deployment but causes rebalancing on app restarts. Fat-client mode ( clientMode=true) isolates data layer; business nodes restart without partition churn, at the cost of one extra network hop.

Spring Cache Abstraction

@Bean
public SpringCacheManager cacheManager(Ignite ignite) {
    SpringCacheManager mgr = new SpringCacheManager();
    mgr.setIgniteInstanceName(ignite.name());
    return mgr;
}

Usage:

@Cacheable(value = "orderCache", key = "#orderId", sync = true)
public Order getOrder(long orderId) { return orderMapper.selectById(orderId); }

@CachePut(value = "orderCache", key = "#order.id")
public Order updateOrder(Order order) { ... }

@CacheEvict(value = "orderCache", key = "#orderId")
public void deleteOrder(long orderId) { ... }

Pitfall: SpringCacheManager creates caches dynamically with limited config (e.g., backups=0). Pre-declare CacheConfiguration and use IgniteCache API directly in production.

Spring Data Repository

public interface OrderRepository extends IgniteRepository<Order, Long> {
    List<Order> findByUserId(long userId);
    @Query("SELECT * FROM \"Order\" WHERE amount > ? AND status = ?")
    List<Order> findBigOrders(BigDecimal amount, int status);
}

IgniteRepository

maps to distributed cache; writes use batch putAll. Note: Order is a SQL reserved word — quote the table name.

Data Access: KV to SQL

Key-Value API

IgniteCache<Long, Order> cache = ignite.cache("orderCache");
cache.put(1L, order);
Order o = cache.get(1L);
cache.putIfAbsent(1L, order);
cache.putAll(Map.of(1L, o1, 2L, o2)); // batched by partition
boolean ok = cache.replace(1L, oldOrder, newOrder); // CAS

Batch operations group keys by partition, merging network requests. Always prefer putAll over looped put — a common performance trap.

SQL Queries

Ignite embeds H2 for ANSI SQL, secondary indexes, aggregations.

public class Order implements Serializable {
    @QuerySqlField(index = true) private long id;
    @QuerySqlField(index = true) private long userId;
    @QuerySqlField private BigDecimal amount;
    @QuerySqlField private String status;
}

CacheConfiguration<Long, Order> cfg = new CacheConfiguration<>("orderCache")
    .setIndexedTypes(Long.class, Order.class);

Query execution:

SqlFieldsQuery qry = new SqlFieldsQuery(
    "SELECT userId, SUM(amount) FROM \"Order\" WHERE status = ? GROUP BY userId")
    .setArgs(1);
try (QueryCursor<List<?>> cursor = cache.query(qry)) {
    for (List<?> row : cursor) {
        System.out.println(row.get(0) + " -> " + row.get(1));
    }
}

Best practices: avoid SELECT * (use SqlFieldsQuery for projection), add setCollocated(true) for parallel Map-Reduce on collocated data, avoid LIKE '%x%' (full partition scan).

Distributed Joins

-- Local join if affinity keys match
SELECT o.id, i.productId
FROM "Order" o JOIN OrderItem i ON o.id = i.orderId
WHERE o.userId = ?

Non-collocated joins trigger broadcast — design affinity keys to keep join partners together.

Near Cache

NearCacheConfiguration<Long, Order> nearCfg = new NearCacheConfiguration<>()
    .setNearStartSize(100_000)
    .setNearEvictionPolicy(new LruEvictionPolicy<>(50_000));

CacheConfiguration<Long, Order> cfg = new CacheConfiguration<>("orderCache")
    .setNearConfiguration(nearCfg);

Client-side only; reads hit local memory first (10× speedup). Trade-off: memory amplification per client, invalidation overhead on writes. Best for read-heavy, hotspot workloads.

Continuous Queries

ContinuousQuery<Long, Order> cq = new ContinuousQuery<>();
cq.setLocalListener(events -> events.forEach(e -> {
    if (e.getEventType() == EventType.PUT) {
        alertService.onOrderChanged((Order) e.getNewValue());
    }
}));
// Remote filter runs on server — only matching changes cross network
cq.setRemoteFilterFactory(() -> (CacheEventFilter<Long, Order>) (type, oldVal, newVal) ->
    newVal != null && newVal.getAmount().compareTo(BIG) > 0);
QueryCursor<Entry<Long, Order>> cursor = cache.query(cq);
// Keep cursor open; closing stops the continuous query

Remote filter is essential — without it, every mutation floods the network.

Distributed Capabilities

Locks & Atomics

// Reentrant distributed lock
IgniteLock lock = ignite.reentrantLock("lock:order:" + orderId, true, true, true);
lock.lock();
try { /* critical section */ } finally { lock.unlock(); }

// Atomic long for counters (each op may hit network)
IgniteAtomicLong stock = ignite.atomicLong("stock:sku:1001", 0, true);
long remain = stock.decrementAndGet();

// High-throughput sequence with batch pre-fetch
IgniteAtomicSequence seq = ignite.atomicSequence("orderSeq", 0, true);
seq.incrementAndGet();

IgniteAtomicLong

becomes a bottleneck at high concurrency; use partitioned aggregation or IgniteAtomicSequence batch pre-fetch instead.

Compute Grid: Move Computation to Data

IgniteCompute compute = ignite.compute();
// Affinity call — runs on node owning the key
OrderScore score = compute.affinityCall("orderCache", orderId,
    () -> localScoreEvaluator.evaluate(orderId));
// Broadcast to all nodes
compute.broadcast(() -> { localCache.clear(); return null; });
// Apply closure to argument collection
Collection<String> result = compute.apply(
    (IgniteClosure<String, String>) s -> s.toUpperCase(),
    List.of("a", "b", "c"));

affinityCall

is the key primitive: guarantees execution on the data node, turning “move millions of rows” into “local in-memory scan” for risk scoring.

Service Grid

ServiceConfiguration svcCfg = new ServiceConfiguration()
    .setName("riskEngine")
    .setService(new RiskEngineImpl())
    .setTotalCount(1)
    .setMaxPerNodeCount(1);
ignite.services().deploy(svcCfg);

RiskEngine engine = ignite.services().serviceProxy(
    "riskEngine", RiskEngine.class, false);

Singleton services auto-failover on node loss. For sharded deployment, increase setTotalCount and avoid mixing with singleton affinity config.

Messaging

IgniteMessaging messaging = ignite.message();
// Point-to-point
messaging.send(ignite.cluster().forNodeId(nodeId), "topic", "hello");
// Topic subscribe
messaging.localListen("topic", (nodeId, msg) -> {
    log.info("Received from {}: {}", nodeId, msg);
    return true; // continue listening
});

High Availability: Failure, Partition Loss & Recovery

Node Failure & Replica Promotion

Heartbeat timeout marks node lost; backup partitions promote to primary automatically. Client retries redirect to new primary — near-transparent. However, replica rebuild takes time; during rebuild the cluster is degraded. A second simultaneous failure risks data loss.

Partition Loss

If both primary and backup for a partition are unavailable, partition loss occurs. Default: cache becomes unavailable. Recovery:

// 1. Confirm dead nodes won't return
// 2. Reset lost partitions (data may be incomplete)
ignite.resetLostPartitions(List.of("orderCache"));

Only with persistence enabled and intact disk data can a restarted node truly recover. backups=0 is the #1 cause of unrecoverable partition loss.

Persistence & Baseline Topology

Persistence requires explicit baseline topology — the set of server nodes participating in persistence and rebalancing.

ignite.cluster().setBaselineTopology(ignite.cluster().forServers().nodes());

Cluster restart waits for all baseline nodes. If a node is permanently gone, remove it from baseline first, else cluster stalls. setAutoAdjustBaselineTopology(true) automates this but manual control is safer for production.

Snapshots

ignite.snapshot().createSnapshot("snapshot-2024-06-01").get();
ignite.snapshot().restoreSnapshot("snapshot-2024-06-01").get();

Copy-on-write snapshots minimally impact online workloads. Incremental snapshot behavior varies by version — verify docs.

Performance Tuning

Off-Heap Memory Allocation

Data lives off-heap (no GC). Set -Xmx small (4–8 GB), give rest to off-heap. Multiple data regions allow tiering: hot region large + LRU eviction; cold region small + persistence.

DataRegionConfiguration hot = new DataRegionConfiguration()
    .setName("hotRegion")
    .setMaxSize(8L * 1024 * 1024 * 1024)
    .setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU)
    .setEvictionThreshold(0.9);

new CacheConfiguration<>("hotCache").setDataRegionName("hotRegion");

Page Size

Default 4 KB; options 1/2/4/8/16 KB. Small objects + random access → 4–8 KB. Large objects + sequential scans → larger pages. Persistence often locks to 4 KB. Page size is immutable after cluster initialization.

Indexing

Index only columns used in WHERE / JOIN. Indexes consume memory and slow writes. Composite indexes follow leftmost-prefix rule. Use EXPLAIN to verify Index Scan vs Full Scan. SqlQuery with partitions parameter can limit scan scope.

Concurrent Updates

Two atomicity modes: ATOMIC (per-operation, no tx overhead, best for caching) and TRANSACTIONAL (ACID, cross-key consistency, higher cost). For high-contention counters, use EntryProcessor to push read-modify-write to the data node:

cache.invoke(orderId, (entry, arg) -> {
    Order o = entry.getValue();
    o.setAmount(o.getAmount().add((BigDecimal) arg));
    entry.setValue(o);
    return null;
}, delta);

Network & Serialization

Disable peerClassLoading in prod (security, non-determinism). For partial reads, use BinaryObject with withKeepBinary() to avoid full deserialization:

IgniteCache<Long, BinaryObject> binCache = cache.withKeepBinary();
BinaryObject bo = binCache.get(1L);
String status = bo.field("status"); // no full object deserialization

Custom BinaryMarshaller requires strict field-order management via BinaryNameMapper. Tune TcpCommunicationSpi socket send buffer and message queue limits alongside GC/network monitoring.

Selection & Deployment

Vs Redis, Caffeine, Hazelcast

Caffeine: single-node hot data, no sharing/persistence/SQL. Redis: shared cache, limited ops, weak on complex queries, hot keys, compute-near-data. Hazelcast: overlaps with Ignite; choose if team knows IMDG and SQL not needed. Ignite wins on unified memory+disk storage, full SQL, strong affinity scheduling, compute grid. Cost: operational complexity (baseline topology, partitions, persistence, snapshots, memory regions).

Production Monitoring

JMX exposes ignite.metrics(). Prometheus via ignite-prometheus module. Visualize with Ignite Web Console or GridGain Control Center. Critical metrics: CacheHitPercentage, CachePutsPerSec, PartitionCount, RebalancingPartitionsCount, OffHeapSizeUsed, WalSegmentsCount — early signals for rebalancing, off-heap pressure, WAL backlog.

Pitfall Checklist

backups=0

— major risk, especially with persistence. Minimum 1 backup in prod.

Client node misconfigured as clientMode=false — business restarts trigger massive rebalance.

Persistent cluster without baseline topology — partitions won't recover on restart. @Cacheable dynamic cache creation — no indexes, no replicas, config drift.

Large objects (hundreds of KB JSON) — page fragmentation, serialization overhead. Trim fields first.

Looped put instead of putAll.

IP finder port range (47500–47509) not fully opened in security groups.

Ignite upgrade ignoring binary compatibility — BinaryMarshaller field-order changes break deserialization. Snapshot + canary first.

Server/client JVM params misaligned (container memory limits, off-heap, data region maxSize) → OOMKill.

Continuous query without remote filter — network drowned in change events.

Adoption Rhythm

Phase 1: Fat-client mode against existing server cluster, caching only, backups=1 + near cache, validate plumbing & monitoring. Phase 2: Add SQL indexes & affinity design, migrate complex queries & some JOINs. Phase 3: Enable native persistence & snapshots, promote core data to system-of-record. Phase 4: Push batch logic into compute grid. Incremental adoption turns Ignite into a reliable second database, not another opaque middleware.