Building a User Profiling Platform with Spring Boot: Tags, Rules & Real-time Updates

1. Business Background

The platform solves a concrete problem: transaction, behavior, content, and service systems each have separate user data, forcing businesses to rewrite SQL for every audience selection. The platform organizes these signals into tags and lets operations and systems consume by tag. In e-commerce, content, SaaS, and finance scenarios, the platform handles three workloads:

Precision Marketing : Filter high-potential users by "GMV last 30 days > 1000 and active within 7 days" and push to marketing reach systems.

Audience Selection : Operations combine tag conditions to get exportable, fixed audience packages within minutes.

Personalized Operations : Client-facing APIs return profiles by userId to drive recommendations, benefits, pop-ups, and tiered pricing.

Before platformization, each business line wrote its own SQL, scheduled scripts, and local Excel files. Pain points included inconsistent definitions, high development cost (one-week lead for a new tag), poor real-time performance (T+1 offline), uncontrollable selection (large joins taking tens of minutes), and no governance (lineage, quality, access audits were black boxes).

This platform is built on a Spring Boot microservice family covering tag definition, rule computation, tag storage, audience selection, real-time updates, version governance, and security compliance.

2. Platform Architecture

2.1 Layered Architecture

graph TB
    subgraph Access Layer
        A1[Admin Console]
        A2[OpenAPI / SDK]
        A3[Marketing System / Recommendation System]
    end

    subgraph Service Layer Spring Boot
        B1[profile-admin Tag Definition Center]
        B2[profile-query Profile Query Service]
        B3[profile-crowd Audience Selection Service]
        B4[profile-governance Governance Center]
        B5[profile-gateway Gateway / Auth]
    end

    subgraph Compute Layer
        C1[Real-time Compute Kafka + Flink/Spring Consumer]
        C2[Offline Compute Spark SQL]
        C3[Rule Engine Aviator]
        C4[Scheduler XXL-JOB]
    end

    subgraph Storage Layer
        D1[(MySQL Metadata)]
        D2[(ClickHouse/Hive Tag Wide Table)]
        D3[(HBase/Redis Profile KV)]
        D4[(Elasticsearch Tag Index)]
        D5[(OSS/RocksDB Bitmap Audience)]
    end

    A1 --> B5
    A2 --> B5
    A3 --> B5
    B5 --> B1
    B5 --> B2
    B5 --> B3
    B5 --> B4
    B1 --> C4
    C4 --> C2
    B1 --> C3
    C1 --> C3
    C1 --> D3
    C2 --> D2
    B3 --> D4
    B3 --> D5
    B2 --> D3

2.2 Technology Stack

The framework uses Spring Boot 3.x + Spring Cloud for service registration, config center, and gateway. Metadata lives in MySQL 8 with MyBatis-Plus, storing tag definitions, versions, audience definitions, and lineage. The rule engine uses Aviator or QLExpress with timeout and sandbox protection to prevent a bad rule from saturating thread pools.

Real-time pipeline: Kafka + Flink, or Spring Boot consumers for simple cases. Offline compute: Spark SQL / Hive for statistical tags and T+1 full recomputation. Tag wide table primarily in ClickHouse, Hive for archival. Single-user query: Redis hot data + HBase full fallback. Tag combination search and exploration: Elasticsearch. Audience bitmaps: RoaringBitmap serialized to Redis or OSS. Scheduler: XXL-JOB. Auth and audit: Spring Security + RBAC + audit logs, with tenant and tag-level ACLs.

2.3 Service Responsibilities

profile-admin : Tag creation, editing, version publishing, deprecation. Writes only metadata, no computation.

profile-query : Unified query entry for client and internal use, returns userId -> {tagCode: value} with multi-level caching.

profile-crowd : Audience condition parsing, bitmap operations, estimation, export, snapshots.

profile-governance : Lineage parsing, quality inspection, heat statistics, cold/hot migration, audit.

profile-gateway : Unified auth, tenant identification, rate limiting, desensitized responses.

Design principle: decouple definition from computation. Tag definition is just metadata; the compute engine reads metadata and routes by calc_mode to real-time or offline pipelines. Adding a tag requires no code deployment, preventing the platform from degrading into a "ticket-per-tag" bottleneck.

3. Tag Modeling

3.1 Four Tag Types

Fact tags directly map source fields without processing (e.g., registration channel, gender, city, membership level), usually synced or near-real-time extracted. Rule tags are determined by conditional expressions, e.g., high-value user = gmv_30d > 1000 && order_cnt_30d >= 3, evaluated by the expression engine. Statistical tags are time-window aggregations (e.g., active days last 7 days, 30-day average order value, cumulative order count), computed via SQL aggregation or streaming windows. ML tags are model outputs (scores or buckets), e.g., churn probability, LTV prediction, category preference Top3, produced by offline model scoring services.

Additionally, derived tags are second-order computations from other tags, e.g., user_value_tier = f(high_value_tag, active_tag). Derived tags naturally form a lineage DAG, which is a governance focus.

3.2 Metadata Model

CREATE TABLE tag_def (
  id BIGINT PRIMARY KEY AUTO_INCREMENT,
  tenant_id VARCHAR(32) NOT NULL,
  tag_code VARCHAR(64) NOT NULL COMMENT 'unique within tenant',
  tag_name VARCHAR(64) NOT NULL,
  tag_type VARCHAR(16) NOT NULL COMMENT 'FACT/RULE/STAT/MODEL/DERIVE',
  value_type VARCHAR(16) NOT NULL COMMENT 'ENUM/NUMBER/DATE/STRING/BOOL/SET',
  category VARCHAR(32) NOT NULL COMMENT 'demographic/transaction/behavior/content',
  value_range JSON NULL COMMENT 'enum values / numeric range',
  sensitive TINYINT DEFAULT 0,
  owner VARCHAR(64) NOT NULL,
  status TINYINT DEFAULT 0 COMMENT '0 draft 1 canary 2 online 3 offline',
  current_ver INT DEFAULT 0,
  create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
  UNIQUE KEY uk_tenant_code (tenant_id, tag_code)
);

CREATE TABLE tag_version (
  id BIGINT PRIMARY KEY AUTO_INCREMENT,
  tag_id BIGINT NOT NULL,
  version INT NOT NULL,
  calc_mode VARCHAR(16) NOT NULL COMMENT 'REALTIME/OFFLINE/BOTH',
  rule_expr TEXT NULL,
  source_conf JSON NULL COMMENT 'source table/event/window/agg function',
  schedule_cron VARCHAR(64) NULL,
  status VARCHAR(16) NOT NULL,
  change_log VARCHAR(512) NULL,
  create_by VARCHAR(64) NULL,
  create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
  UNIQUE KEY uk_tag_ver (tag_id, version)
);

Note: tag_code uniqueness is tenant_id + tag_code, not global, to support multi-tenancy.

3.3 Versioning and Release Flow

Once a tag is referenced downstream, changing its definition is an incident. The platform uses versioning + immutability:

DRAFT -> VALIDATING -> CANARY -> ONLINE -> OFFLINE

Drafts are mutable; online versions are frozen. Any change creates a new version. Canary releases affect 5% of users, comparing new vs. old distribution; PSI exceeding threshold triggers auto-rollback. During early online phase, dual-write old and new versions; query side reads by tenant + tag + version. After observation period, traffic switches.

@Service
@RequiredArgsConstructor
public class TagPublishService {
    private final TagVersionMapper versionMapper;
    private final TagValidator validator;
    private final CanaryService canaryService;
    private final ApplicationEventPublisher eventPublisher;

    @Transactional(rollbackFor = Exception.class)
    public void submitCanary(Long tagId, int version) {
        TagVersion v = versionMapper.selectByTagIdAndVersion(tagId, version);
        validator.validate(v); // syntax, field existence, value range
        versionMapper.updateStatus(v.getId(), "CANARY");
        // external canary action not in transaction; publish event for post-commit execution
        eventPublisher.publishEvent(new CanaryStartEvent(tagId, version, 0.05));
    }

    public void promote(Long tagId, int version) {
        canaryService.assertPsiWithinThreshold(tagId, version);
        versionMapper.updateStatusByTagAndVersion(tagId, version, "ONLINE");
    }
}

4. Computation Pipelines

4.1 Real-time Event Ingestion

User behaviors (view, add-to-cart, order, payment, login) converge via Kafka. Spring Boot consumers handle lightweight real-time tags. A key pitfall: counter keys should be bucketed by day, and expiration set only on first incr. Calling expire on every increment pushes TTL forward, breaking window counts.

@Slf4j
@Component
@RequiredArgsConstructor
public class BehaviorListener {
    private final StringRedisTemplate redis;
    private final TagRealtimeService tagService;

    @KafkaListener(topics = "user-behavior", groupId = "profile-realtime", concurrency = "8")
    public void onEvent(String payload) {
        UserEvent e = JSON.parseObject(payload, UserEvent.class);
        // bucket by day to avoid cross-day count mixing
        String day = LocalDate.now().toString();
        String key = String.format("stat:%s:%s:%s", e.getUserId(), e.getEventType(), day);
        Long n = redis.opsForValue().increment(key);
        if (n != null && n == 1L) {
            redis.expire(key, 2, TimeUnit.DAYS);
        }
        tagService.refresh(e.getUserId(), e);
    }
}

Spring Boot consumers suit "few operators + low state" real-time tags. Complex stateful scenarios like 30-day sliding window UV should use Flink. Both read the same tag_version metadata to avoid definition drift.

4.2 Offline Batch Computation

Statistical tags are triggered by the scheduler to run Spark SQL / ClickHouse jobs, with SQL generated from configuration:

{
  "tagCode": "gmv_30d",
  "calcMode": "OFFLINE",
  "source": {
    "type": "clickhouse",
    "table": "dwd_order",
    "filter": "status = 'PAID' AND dt >= today() - 30",
    "groupBy": "user_id",
    "agg": "sum(pay_amount)"
  }
}

The engine compiles config to SQL, writes to wide table ads_user_tag_df partitioned by dt, then syncs to ClickHouse and HBase. High-frequency tags use the wide table; full tag set later uses row-oriented storage.

4.3 Rule Expressions

Rule tags use Aviator for performance and Java-like syntax. Sandbox, timeout, and variable whitelist are mandatory to prevent runaway rules.

@Configuration
public class RuleEngineConfig {
    @Bean
    public AviatorEvaluatorInstance ruleEngine() {
        AviatorEvaluatorInstance engine = AviatorEvaluator.newInstance();
        engine.setOption(Options.MAX_LOOP_COUNT, 1000);
        engine.setOption(Options.EVAL_TIMEOUT_MS, 50); // 50ms per rule
        // ASM compile switch per Aviator version; don't hardcode enum in article
        return engine;
    }
}

@Slf4j
@Service
@RequiredArgsConstructor
public class RuleTagEvaluator {
    private final AviatorEvaluatorInstance engine;

    public boolean match(String expr, Map<String, Object> ctx) {
        try {
            Expression exp = engine.compile(expr, true); // enable compile cache
            return Boolean.TRUE.equals(exp.execute(ctx));
        } catch (Exception ex) {
            log.warn("rule eval failed, expr={}, ctx={}", expr, ctx, ex);
            return false; // default no match, but metric emitted
        }
    }
}

Rule example:

gmv_30d > 1000 && order_cnt_30d >= 3 && last_active_days <= 7

. Context ctx is batch-fetched from KV store; rule execution is sharded by user with thread-pool isolation. With simple rules and good context prefetch, single-node throughput of thousands to tens of thousands TPS is possible; complex rules or KV latency spikes change numbers dramatically — don't treat TPS as a fixed benchmark.

4.4 Tag Backfill

When definitions change or upstream data is fixed, backfill by date is needed. Key points: partition idempotency, rate limiting, resume from failure. Wide table partitioned by dt — single-tag backfill cannot delete the whole partition (would wipe other tags of that day). Options: full table recompute, or use row-oriented detail table keyed by tag_code + version + dt for targeted deletion.

public class BackfillJob {
    public void backfill(String tagCode, int version, LocalDate start, LocalDate end) {
        for (LocalDate dt = start; !dt.isAfter(end); dt = dt.plusDays(1)) {
            // row-oriented detail table: idempotent delete by tag, version, date
            clickHouse.delete("ads_user_tag_detail",
                "dt = {dt:Date} AND tag_code = {tag:String} AND version = {ver:UInt32}",
                dt, tagCode, version);
            computeAndWrite(tagCode, version, dt);
        }
    }
}

For wide table ads_user_tag_df, safer to recompute the day's full partition or use a temp table then REPLACE PARTITION. Avoid deletePartition shortcuts.

4.5 Quality Checks

After each computation, automated quality checks run; failures block release. Common checks:

Null rate > 30% → alert, no release.

Enum values outside value_range → alert.

PSI vs. previous day > 0.25 → alert and block canary.

Valued-user count / total users drop > 20% MoM → alert. user_id duplicate count > 0 → hard block.

PSI thresholds should be calibrated per tag's historical distribution; 0.25 is lenient, core tags can be stricter.

5. Storage Selection

No single storage fits all; a layered combination is used:

graph LR
    Q[Query Request] --> C1{Scenario}
    C1 -->|Single User Full Profile| R1[Redis Hot Data] --> R2[HBase Full]
    C1 -->|Tag Combination Estimation| R3[Elasticsearch]
    C1 -->|Precise Audience Selection| R4[RoaringBitmap]
    C1 -->|Offline Analysis & Backfill| R5[ClickHouse / Hive]
    C1 -->|Metadata| R6[MySQL]

ClickHouse holds wide table ads_user_tag_df — columnar, high compression, fast aggregation, suitable for offline analysis, coarse audience filtering, and backfill. Hive stores full historical snapshots at low cost for archival and cheap backfill. HBase stores uid -> all tags row-oriented data for random read and horizontal scaling, serving as full-profile query fallback. Redis caches hot user profiles and tag bitmaps for sub-millisecond client queries and selection acceleration. Elasticsearch stores inverted tag indexes for combination search and aggregation counts, useful for selection estimation and tag distribution exploration. RoaringBitmap stores bitmap per tag value; fast set operations and good compression for audience selection and package storage.

Wide table schema example:

CREATE TABLE ads_user_tag_df (
  user_id UInt64,
  gmv_30d Decimal(18,2),
  order_cnt_30d UInt32,
  last_active_days UInt16,
  member_level LowCardinality(String),
  high_value_flag UInt8,
  churn_score Float32,
  dt Date
) ENGINE = ReplacingMergeTree
PARTITION BY dt
ORDER BY (user_id)
SETTINGS index_granularity = 8192;

As tag count grows, avoid infinite column expansion. Practical approach: dual-write — top 200 high-frequency tags in wide table, full tag set in row-oriented user_id + tag_code + tag_value table. Query routing by scenario; don't force all queries to scan row table.

Multi-level cache: Redis (TTL 10min) → HBase (full) → ClickHouse (fallback). Hot user hit rate ~95%; long-tail users still fall back — don't hardcode hit rate as a platform KPI.

6. Audience Selection

6.1 Condition DSL

Operations see a condition tree; backend uses a unified DSL:

{
  "operator": "AND",
  "children": [
    { "tag": "member_level", "op": "IN", "values": ["V3", "V4"] },
    { "tag": "gmv_30d", "op": "GTE", "value": 1000 },
    {
      "operator": "OR",
      "children": [
        { "tag": "last_active_days", "op": "LTE", "value": 7 },
        { "tag": "push_click_30d", "op": "GTE", "value": 3 }
      ]
    }
  ]
}

6.2 Bitmap Set Operations

Core selection maps each "tag=value" condition to a RoaringBitmap, then performs set operations. Two caveats: RoaringBitmap defaults to 32-bit; raw userId as 64-bit snowflake IDs requires either an internal uid_mapping to 32-bit or using Roaring64NavigableMap. AND/OR short-circuit logic differs: OR cannot exit early just because current result is empty.

@Service
@RequiredArgsConstructor
public class CrowdSelectService {
    private final BitmapIndexService bitmapIndex;

    public RoaringBitmap select(ConditionNode root) {
        return eval(root);
    }

    private RoaringBitmap eval(ConditionNode node) {
        if (node instanceof LeafNode leaf) {
            return bitmapIndex.getBitmap(leaf);
        }
        List<ConditionNode> children = node.getChildren();
        if (children == null || children.isEmpty()) {
            return new RoaringBitmap();
        }
        RoaringBitmap acc = eval(children.get(0));
        for (int i = 1; i < children.size(); i++) {
            if (node.isAnd()) {
                if (acc.isEmpty()) {
                    return acc;
                }
                acc = RoaringBitmap.and(acc, eval(children.get(i)));
            } else {
                acc = RoaringBitmap.or(acc, eval(children.get(i)));
            }
        }
        return acc;
    }
}

Bitmap index key: bitmap:{tenant}:{tagCode}:{value}. Serialized RoaringBitmap: hot values in Redis, cold in local RocksDB or OSS. Enum/Set tags can build bitmaps directly by value. Numeric range conditions like gmv_30d >= 1000 require either numeric bucket bitmaps or candidate fetch from ES/ClickHouse then bitmap construction; pure tagCode:value cannot support range queries.

Why not SQL join? On 100M+ users, RoaringBitmap intersections are typically milliseconds to hundreds of milliseconds; sparse scenarios save significant memory vs List<Long>, dense scenarios have stable compression. AND/OR/NOT and cardinality via getLongCardinality() are built-in. NOT requires a full-user bitmap for difference; otherwise impossible.

6.3 Audience Estimation

Before selection, give operations an estimated count to avoid waiting. Indexed tags can compute exact cardinality via bitmap getLongCardinality(). For 100M+ with ±3% tolerance, ES count or HyperLogLog estimation works. Cost optimization: for AND, evaluate smallest-cardinality condition first to short-circuit on empty set; for OR, be careful not to short-circuit and miss branches.

public CrowdEstimate estimate(ConditionNode root) {
    long start = System.currentTimeMillis();
    RoaringBitmap bm = select(root);
    return new CrowdEstimate(bm.getLongCardinality(), System.currentTimeMillis() - start);
}

6.4 Paginated Export and Audience Snapshots

Exporting millions of users cannot pull all at once; uses bitmap sharding + cursor:

public void export(Long crowdId, Sink sink) {
    RoaringBitmap bm = crowdSnapshotService.load(crowdId);
    int[] buckets = bm.toArray(); // assumes user_id mapped to 32-bit internal uid
    int pageSize = 5000;
    for (int i = 0; i < buckets.length; i += pageSize) {
        int end = Math.min(i + pageSize, buckets.length);
        List<Long> ids = IntStream.range(i, end)
            .mapToObj(k -> (long) buckets[k])
            .toList();
        List<UserProfile> rows = profileQuery.batchGet(ids); // enrich with tag details
        sink.write(rows);
    }
}

Snapshots solve "selection changes after creation". Marketing needs a fixed audience, not real-time recomputation each time.

public CrowdSnapshot createSnapshot(Long crowdId) {
    ConditionNode cond = crowdDefinitionService.load(crowdId);
    RoaringBitmap bm = crowdSelectService.select(cond);
    byte[] bytes = BitmapCodec.serialize(bm);
    String path = oss.put("crowd/" + crowdId + "/" + snapshotId + ".rb", bytes);
    return snapshotMapper.insert(CrowdSnapshot.builder()
        .crowdId(crowdId)
        .size(bm.getLongCardinality())
        .storagePath(path)
        .createTime(LocalDateTime.now())
        .build());
}

Snapshots are versioned: same audience can generate v1/v2/v3. Marketing jobs reference a fixed version for reproducibility and traceability. Don't skip version numbers; they are essential for debugging "why this reach audience differs from last time".

7. Updates and Governance

7.1 Real-time Updates

Real-time pipeline: behavior events → Kafka → real-time compute updates Redis/HBase → emit tag change events to downstream. Consistency: write primary HBase first, then invalidate cache (Cache-Aside). Concurrent reads refilling stale values are handled by version numbers or delayed double-delete. Real-time tags tolerate second-level latency; for extreme consistency (e.g., account balance), query source system directly — don't rely on profile.

7.2 Version Management

Tag versions control definitions; audience versions control results. All references recorded in tag_reference table; before deprecating a tag, check reference count. Platform should provide diff tool: run same user batch through two versions, output divergence report — don't let business guess.

7.3 Data Lineage

Lineage auto-generated by parsing compute configs, not manual. Example:

Event dwd_user_behavior --+
Order table dwd_order ------+--> Tag gmv_30d --> Tag high_value --> Audience HighPotentialUserA

Stored as graph (MySQL adjacency or Neo4j). Capabilities: impact analysis (which tags affected by table change), root cause (upstream of anomalous tag), deprecation assessment.

7.4 Quality Monitoring

Daily inspection produces quality score, fed to dashboard and alerts. Metrics: null rate, coverage, enum violation rate, distribution PSI, compute latency, job success rate. Alert strategy: two consecutive cycles of degradation → alert; severe anomalies auto-pause downstream dependent tasks to stop dirty data propagation.

7.5 Cold/Hot Eviction

Tag usage heat from query logs aggregated to tag_query_stat. Hot tags (>100k daily queries) → Redis + ClickHouse wide column. Warm tags (thousands daily) → HBase row store. Cold tags (<100 monthly) → downgrade to Hive, retain only backfill capability; 90 days no access → auto-offline and notify owner. Deprecation still passes reference check.

8. Security and Implementation

8.1 Tenant Isolation

All tables carry tenant_id; MyBatis interceptor auto-injects tenant condition to prevent cross-tenant queries. Resource layer: large tenants get dedicated ClickHouse databases or Redis prefixes to avoid noisy neighbors. Task layer: per-tenant quota rate limiting. ClickHouse, HBase must also enforce tenant dimension — not just MySQL.

8.2 Field Permissions and Desensitization

RBAC + tag-level ACL dual control. L1 public tags (member level, city) visible to all tenants. L2 restricted tags (spend amount, activity) require authorization with audit trail. L3 sensitive tags (phone, ID, address) default masked; plaintext requires secondary approval + watermark + export limits.

public String mask(String tagCode, String value, UserPrincipal user) {
    if (!tagAclService.canReadPlain(user, tagCode)) {
        return DesensitizeUtils.mask(value); // 138****8888
    }
    auditService.record(user, tagCode, "READ_PLAIN");
    return value;
}

All plaintext queries and audience exports logged: who, when, what, how many. Compliance audits rely on these logs more than documentation.

8.3 Performance Testing

Pre-launch stress tests for key scenarios. Single-user profile query with Redis hit: P99 < 10ms, QPS > 20k. ClickHouse fallback P99 ~100ms. 100M-user dual-condition selection with warmed bitmaps: P99 < 500ms. Five-level nested complex selection: P99 < 2s realistic. Million-user export via async job: < 5 minutes. Real-time event processing: 5k TPS per node, end-to-end latency < 3s (depends on event size and downstream writes). Watch bitmap load time, Redis big keys, ClickHouse concurrent query saturation, rule engine cache hit rate — these are the first bottlenecks.

8.4 Implementation Roadmap

MVP (1-2 months): tag definition + offline statistical tags + basic audience selection (bitmap) + single-user query. Enhancement (2-3 months): real-time tags, rule tags, version canary, quality checks, export & snapshots. Governance (1-2 months): lineage, audit, permission masking, cold/hot eviction, performance tuning. Intelligence (ongoing): ML tag integration, tag recommendation, lookalike expansion, automated ops loop — pace by business rhythm, not all at once.

8.5 Common Pitfalls

First, don't launch with full tag set. Validate high-frequency, stable-definition tags first; verify wide-table and row-store dual-write pipelines before expanding tag count. Second, bitmap indexes need warming and sharding; Redis big keys are inevitable — hot tag bitmaps in local cache or value-sharded, don't wait for saturation. Third, versioning and audit cannot be retrofitted. Tag definition changes, audience exports, plaintext queries — late governance equals rebuild. Fourth, don't force real-time pipeline to handle long windows; Spring Boot consumers for lightweight increments, complex state to Flink — saves many late-night debugging sessions.