Enterprise Codex Governance: Trustworthy Collection, Secure Access, Auditable Lifecycle

Background: Enterprises Need Trustworthy Collection, Not Just More Data

When Codex moves from individual experimentation to enterprise-scale deployment, organizations face new questions: plugin coverage across employees, device health, sync timestamps, data loss during network outages, duplicate uploads, admin overreach, and post-employment data retention. Traditional log collection fails because Codex data originates on employee machines with unstable networks, evolving file formats, and inherent context (employee, device, project, session). Misattribution causes cross-employee data leaks.

The project establishes three boundaries:

Collect only employee-authorized, client-readable, protocol-supported data — no full-disk scans, unauthorized directories, clipboard, browser history, or personal chats.

Never infer identity from OS usernames, directory names, or content. Attribution must come from enterprise identity, one-time activation codes, device credentials, and server-verified trusted context.

Admins default to L0/L1 metadata (run status, aggregates). Desensitized (L2) and raw (L3) content require independent approval, short-lived grants, and full audit trails — no super-admin bypass.

The core problem: make every record answer who authorized it, where it came from, what processing it underwent, its current state, and who accessed or disposed of it.

System Architecture: End-to-End Loop from Employee Machine to Admin Console

Local Collection & Employee Control

Codex lifecycle hooks capture session start, user submit, tool call, turn end, session end. Hooks do not upload directly; they hand events to a local resident Agent within a 90 ms budget (25 ms for Unix socket connect). The Agent writes to an encrypted SQLite queue (WAL mode, strict sync). Payloads encrypted per-record with AES-256-GCM; queue master key stays in macOS Keychain. Directory, database, and socket permissions set to 0700, 0600, 0600 respectively. Disk quota exhaustion pauses collection explicitly, preserving unacknowledged records — no silent overwrites.

Agent supports initial full sync and continuous incremental sync. New events during full sync go to incremental buffer; after snapshot success, resume from frozen cursor — eliminating the "scan-window data gap." Network interruption, process restart, or shard upload failure reuse original package ID, idempotency key, and upload session for recovery.

Employees can view collection status, pause/resume, trigger immediate sync, regenerate full sync, and bind devices via one-time activation codes. Access tokens, refresh tokens, and device private keys stored only in macOS Keychain — never in plain config, logs, or command args.

Server Ingestion, Parsing & Projection

Four deployment units collaborate via OpenAPI, Kafka events, and explicit app ports: ingest-api: device auth, upload sessions, object validation, raw package persistence to private object storage. parser-worker: defensive unpacking, protocol selection, standardization, atomic commit. Pure transformer — no arbitrary host file access, external network, DB, or system credentials. Unknown source formats go to UNSUPPORTED; raw package retained for future compatible parser. projection-worker: employee summaries, daily metrics, session summaries, governance tasks. control-api: sole query/control entry for admin console.

Raw packages land in private object storage; standardized records and state in MySQL 8.0; Kafka passes minimal lifecycle events. Ingestion layer continues accepting new packages (within quota) even if parser is down — backend faults don't propagate to employee machines.

Package state, state history, and Outbox committed in same transaction. Kafka at-least-once delivery; consumers first write consumed_event. Same event + same digest = safe replay; same event ID + different digest = consistency error — never silent overwrite.

Admin, Audit & Data Governance

Admin console (Ant Design Pro) covers employees, departments, devices, data sources, packages, parse tasks, policies, anomalies, audit logs, activation codes, controlled content, and data governance. Admins see per-employee device trust, heartbeat, backlog, policy application, package state, and full sync command chain from dispatch to success/failure.

Content access layered:

L0/L1: status and metadata (default admin view).

L2: desensitized titles, summaries, standardized records — requires at least one independent approver.

L3: raw content — stays in KMS-encrypted object storage; requires two different admins to approve, Grant max 300 seconds, single-use. On successful read, Grant consumption, access count, and audit commit in same transaction; any failure blocks payload return.

Export, delete, legal hold, and content access all need independent permission, reason, approval, audit — no single-button bypass.

Current schema: 65 MySQL tables, no foreign keys. Consistency via app transactions, unique indexes, state machines, deletion orchestration, automated patrols. Employee deletion splits into 10 re-entrant steps; each step re-checks legal_hold; final verification hash generated only after MySQL and object storage employee scope both zero.

Core Technologies: Reliability, Security, Governance Simultaneously

90 ms Hook & Local Reliable Queue

Hook budget 90 ms total (25 ms socket connect). Does only input upper-bound validation, stable event ID generation, local delivery. Scanning, desensitization, compression, encryption, network upload all in background. Agent returns ACK only after encrypted SQLite transaction commits. Socket temporarily unavailable? Same event envelope retries with same event ID via SQLite short transaction — avoids loss and avoids duplicate records from retries.

Whitelist, Desensitization & Envelope Encryption

Before leaving device: field whitelist, path relativization, credential detection, enterprise sensitive rules. Actions: DROP, MASK, BLOCK. Rule hits log only rule ID and count — never raw sensitive values.

Filtered content packaged as manifest.json + JSONL + blobs, compressed to tar.zst. Each package encrypted with random AES-256-GCM data key; data key wrapped by server KMS public key; SHA-256 of ciphertext computed.

Server re-validates: ciphertext size, digest, Manifest, file set, per-file hash, record count, identity scope. Rejects absolute paths, directory traversal, symlinks, zip bombs, oversized files, unknown fields. Only after server confirms STORED state does local advance uploaded cursor.

Four Deployment Units (Not Over-Split Microservices)

Deliberately kept four units: ingest-api, control-api, parser-worker, projection-worker. Avoids microservice sprawl while maintaining clear boundaries.

Tenant-Employee Isolation & Tiered Authorization

Primary isolation key: tenant_id + employee_id — threads through devices, data sources, packages, standardized records, cache keys, object paths, audit context.

Server never trusts tenant identity from request Body, Query, Path, Manifest, or Kafka Header. Repository must receive trusted scope; employee-data SQL explicitly carries tenant and employee conditions; unauthorized access returns 404 + security audit log.

Use Cases: Who Actually Needs This System

Enterprise Deployment Coverage & Health Operations

R&D platform teams view plugin coverage, version, heartbeat, queue depth, policy version, last sync time per employee/department/device. Quickly identify "installed but not running", "device online but policy not applied", "persistent backlog".

Data Package Troubleshooting & Parse Operations

On upload/validation/parse failure, ops drill from employee → data source → package → parse attempts. See stable error codes, retry counts, sequence gaps, processing state. After parser upgrade, historical raw packages can be re-parsed under audit.

Security & Compliance Audit

Security teams manage collection whitelists, sensitive rules, retention periods, canary policies. View device attestations, policy receipts, content access logs, admin operation audits. Approvals, Grants, exports, deletes, legal holds each have independent state machines and evidence chains.

Controlled Content Viewing

Debugging specific sessions: regular admins see only L0/L1. Post-approval, designated personnel view L2 desensitized content within limited fields, count, time window. L3 raw records require dual-admin approval, single-use grant, short-lived watermark display.

Employee Onboarding, Suspension & Offboarding Governance

One-time activation codes bind employee to device. Suspension: simultaneously disable device, revoke credentials/activation codes, cancel pending tasks. Re-enable does not restore old trust — requires re-bind. Offboarding deletion passes approval, recovery window, legal hold check, zero-residue verification.

This helps AI-heavy enterprises transfer previous employee's AI context quickly to the next handover person.

Small-Scale Enterprise Pilot

Docker Compose single-node pilot config, enterprise OIDC, SCIM-style directory sync, macOS Universal Binary, Developer ID signing, notarization, MDM install templates. Suitable for 5–20 person validation of real network, permissions, ops, upgrade flows.

Roadmap: From Code Complete to Pilot to Platform

Current completion: local collection, device binding, sharded upload, defensive parsing, standardized projection, employee management, content approval, data governance.

Phase 1: Controlled Pilot

Integrate real enterprise IdP & employee directory, production KMS, private object storage, Apple enterprise signing. Validate install, activate, sync, backlog recovery, policy canary, alerting, rollback with 5–20 employees.

Phase 2: Production Operability

Add SLO dashboards, capacity baselines, backup/restore drills, key rotation, runbooks, data governance playbooks, Collector version canary evidence from 1% → 5% → 25% → 100%.

Phase 3: Platform Expansion

Incremental Windows/Linux support, richer data source adapters, project catalog, Git & IDE events. Maintain parser purity, unknown-format fail-closed, employee authorization boundaries unchanged.

Phase 4: Data Value Exploration

Within authorization/privacy boundaries: data quality scoring, search, team reports, management insights.

The system ultimately aims to codify an enterprise AI tool governance method: local collection non-intrusive, data flow verifiable, employee boundaries sharp, sensitive access approved, deletion and retention both evidenced.