Skip to content
From 51 to 14 Days: How Agentic SOC's Multi-Agent Loop Drives Self-Evolution

From 51 to 14 Days: How Agentic SOC's Multi-Agent Loop Drives Self-Evolution

Agentic SOC, a multi-agent security operations product, reduced delivery cycles from 51 to 14 days by establishing a self-evolution loop of real-world failures, unified evaluation (SOCBench with 211 tasks), defined agent autonomy boundaries, and restructuring teams around shared customer outcomes with end-to-end ownership.

Alibaba Cloud Native
Alibaba Cloud Native
Alibaba Cloud Native

01: Starting from Security Operations Tasks

Agentic SOC is a security operations product centered on multi-agent collaboration. Facing alerts scattered across multi-cloud and multi-product environments, it first merges information into investigable and actionable security events, then assigns different agents to handle investigation, assessment, response, and blocking, while also supporting threat hunting and daily security operations.

For users, seeing an analysis report is only part of the process. More important is: what happened, where is the evidence, which objects were affected, and what actions should be taken next. Reports, timelines, and remediation recommendations must jointly answer these questions.

Figure 1: Agentic SOC merges multi-source alerts into security events, with multiple agents collaborating on investigation and response.
Figure 1: Agentic SOC merges multi-source alerts into security events, with multiple agents collaborating on investigation and response.

Therefore, product improvement must return to real tasks. What problems are exposed online, which capabilities fall short, whether modifications solve the problem, and whether they affect previously correct scenarios — all need verifiable evidence.

The team gradually formed a continuous iteration loop: production issues become failure samples, evaluation helps locate capability gaps, agents submit candidate changes, verification and approval decide whether changes take effect. New production results then enter the next round of improvement.

This loop depends on two foundations: a stable standard to judge whether the product has improved, and people who continuously own the final outcome.

02: How a Role Identification Error Became a Regression Asset

In one investigation, the actual event was A attacking B, but the report reversed the roles. Swapping attacker and victim affects the reader's understanding of the entire event. Worse, if multiple modules judge independently, even if the report's roles are corrected, the timeline or remediation advice may still follow another conclusion.

Root cause analysis revealed three modules lacked shared adjudication. Following this cause, the fix expanded from "correcting a report segment" to a structural refactor: unify generation of attacker, victim, and adjudication result fields, then let the report, timeline, and remediation advice reuse these fields.

Figure 2: From error identification, root cause localization, actual modification, to retesting, production effectiveness, and continuous regression — one defect becomes verification basis for subsequent versions.
Figure 2: From error identification, root cause localization, actual modification, to retesting, production effectiveness, and continuous regression — one defect becomes verification basis for subsequent versions.

During retesting, the team reran the original scenario, checking whether A was identified as attacker and B as victim, while verifying that all three modules produced consistent outputs. This confirms the modification fixed the underlying mechanism, not just a single expression.

The role fields and reading logic were then promoted to the formal version, and corresponding regression test cases were added to the evaluation baseline. Future changes to models, prompts, or rules must continue to use this sample to verify role adjudication and cross-module consistency.

One fix leaves two results: the effective product capability, and a repeatable verification sample. The latter frees the team from relying on someone remembering the issue, and gives subsequent improvements a comparable baseline.

This is what the team calls product self-evolution: real problems continuously enter the improvement pipeline, modifications take effect after evaluation and risk judgment, and failure experiences become constraints for the next round. It requires traceable modifications, reliable verification, and clear release boundaries.

03: Using Unified Evaluation to Judge Every Modification

Fixing one case does not prove the overall version improved. A change may solve the role identification problem but cause regression in other scenarios. Therefore, candidate changes must be compared against the same baseline.

Around the security investigation task, the team built SOCBench . At the stage recorded in this sharing, it contains 211 real investigation tasks, covering 16 event types, evaluated from 9 dimensions across 5 investigation stages. Test sets, execution configurations, and scoring specifications are all under version control to enable result reproduction and version comparison.

Figure 3: Evaluation covers different event types and investigation stages; test sets, configurations, and scoring specifications are uniformly version-controlled.
Figure 3: Evaluation covers different event types and investigation stages; test sets, configurations, and scoring specifications are uniformly version-controlled.

Security investigation is a long-horizon task; checking only the final conclusion misses many issues. The team also examines whether environment localization is accurate, entry points confirmed, attack chains reconstructed, impact scope examined, and final conclusions supported by evidence.

Corresponding to capability assessment, beyond report completeness, they check log coverage, time accuracy, entry reconstruction, process chain reconstruction, impact expansion, evidence quality, reasoning and grading, as well as efficiency and stability. This lets evaluation explain "which link needs improvement" rather than just giving a total score.

▍ Simultaneously Evaluating Investigation Quality and Execution Cost

Long-task effectiveness must be understood in conjunction with query volume and execution latency. More queries do not necessarily mean more thorough investigation; shorter latency does not mean the task is complete.

Figure 4: Left side compares version capability profiles; right side observes query calls and stage scores in long tasks.
Figure 4: Left side compares version capability profiles; right side observes query calls and stage scores in long tasks.

In one long-task case shared, Agentic SOC-Latest scored 86 with 43 queries in 28 minutes. A comparison item scored 93 with 95 queries in 48 minutes. This case helps observe the relationship between quality, query overhead, and time; it is a single-case performance, not a direct average for all tasks.

From candidate modification to release, the focus is: whether the original problem passes retest, whether the agreed evaluation scope shows regression, and whether risk is acceptable. Beyond defect-sample retesting, independent test-set evaluation, production task replay, and canary verification are combined. Unapproved modifications are sent back for adjustment, and exposed failure samples continue to be沉淀 into the regression set.

Unified evaluation thus carries three jobs: locate weaknesses, verify improvements, continuous regression. Every change should leave results that are explainable, reviewable, and reusable.

04: Letting Agents Work Autonomously Within Clear Boundaries

As agents take on more steps, human involvement must change. Micromanaging every operation turns humans into task dispatchers and result movers; fully letting go fails to meet security product requirements for production changes.

The team's approach: define four categories of boundaries upfront, letting agents complete analysis, modification, and verification within those boundaries.

Figure 5: Humans pre-define goals, quality, permissions, and stop conditions; agents advance tasks within the agreed scope.
Figure 5: Humans pre-define goals, quality, permissions, and stop conditions; agents advance tasks within the agreed scope.

Goal Boundary — Answers what problems are allowed to be solved. Task goals, priorities, input scope, and expected deliverables must be explicit; work beyond scope requires re-confirmation.

Quality Boundary — Answers what state counts as done. The original issue must pass retest, the agreed evaluation scope must not regress, and definite conclusions cannot be output when evidence is insufficient.

Permission Boundary — Answers what data, tools, and environments can be accessed and changed. High-risk operations must stay within pre-authorized scope; production changes must be traceable and rollbackable.

Stop Boundary — Answers when to hand off to humans. Exceptions, privilege escalation, key metric degradation, retry or resource consumption limits, or inability to make reliable judgments should all trigger a pause.

The clearer these constraints, the clearer the space for agents to advance autonomously. Human energy then concentrates on goal setting, critical judgments, and risk handling. The expected evolution path: from assisted execution to autonomous completion of single-step tasks, to completing the self-evolution loop within boundaries.

05: Turning Cross-Role Handoffs into Shared Result Ownership

After the product internally began continuous iteration around task results, another organizational pattern became prominent: product finishes requirements, security finishes rules, development finishes implementation, testing finishes acceptance. Each segment has deliverables, but the customer's problem is only solved when the whole chain finishes.

At this point, merely speeding up one role's execution compresses total time only slightly. Goals are reinterpreted at each handoff, different roles queue separately, acceptance concentrates at the end, and problems may travel back along the original path.

A scheduling comparison for one requirement illustrates the difference. In the segmented handoff mode, clarification, design, security review, backend, frontend, and centralized acceptance proceeded sequentially, totaling 51 days . In the alternative mode, 12 acceptance criteria were fixed upfront, a single result owner ran the entire process, organizing product experience, engineering delivery, and security effectiveness as parallel tasks, forming a 14-day cycle improvement.

Figure 6: 51-day segmented schedule vs. 14-day parallel cycle.
Figure 6: 51-day segmented schedule vs. 14-day parallel cycle.

In this comparison, the position of acceptance and responsibility is key. Standards that used to be judged centrally at the end were moved to the task start; responsibility that used to transfer with each deliverable is now held by one Owner — the result owner — from goal confirmation all the way to result verification.

Around this working style, organizational collaboration can be summarized into four requirements:

Unified Customer Outcome — Product, security, and engineering must jointly understand what this work improves and what real usage effect judges completion.

Unified Result Standard — Metrics and evidence criteria are fixed upfront, avoiding each role completing its own checklist only to discover misaligned goals at the end.

End-to-End Result Ownership — The owner continuously follows up on issues, dependencies, and verification results, driving task convergence. Parallel work can be done by multiple people and agents, but the result still needs one person accountable.

Risk-Based Expert Involvement — High-risk and irreversible changes must be gated by domain experts; other links participate as needed, reducing unnecessary waiting.

These requirements also explain why team merging and role fusion become important steps. Sharing one goal and completion standard must be implemented in daily division of labor.

06: After Role Fusion, How Professional Capabilities Play Their Part

In this practice, role changes mainly manifest as expanded responsibility scope: looking forward to understand the problem, looking backward to track delivery, while leveraging agents to complete more execution work.

Product Managers extend from PRDs and prototypes into implementation, able to deliver partial frontends and runnable prototypes , using running results to validate product judgments.

Security Engineers expand from focusing solely on security capabilities into product design and implementation , converting professional experience into scenarios, adjudication standards, and effect verification.

Engineers on one hand extend to full-stack closed loops , reducing waiting from separate front-end/back-end scheduling and integration; on the other hand extend toward customer and business understanding , grasping the rationale, value, and commercial impact behind requirements.

Figure 7: Product, security, and engineering expand responsibility scope; whether executed by humans or agents depends on standard clarity and risk.
Figure 7: Product, security, and engineering expand responsibility scope; whether executed by humans or agents depends on standard clarity and risk.

Here it is necessary to distinguish "being able to do more work" from "everyone does all work." For tasks with clear standards, evaluatable, and rollbackable, agents can execute with human approval; for cross-domain but risk-controllable work, the result owner leverages agents to close the loop; high-risk, high-ambiguity, or irreversible work still requires domain experts to lead.

Therefore, professional capabilities remain important, but increasingly manifest in defining problems, judging solutions, verifying effects, and bearing responsibility. Security engineers need to express experience as system-executable, checkable rules and code; product managers need to judge whether runnable results solve real problems; engineers must balance implementation quality and production reliability.

All three roles also share capabilities to strengthen: understanding business and value, and orchestrating AI to complete complex work. How tasks are split, what can run in parallel, what information must be provided, which results must be double-checked — all affect final delivery quality.

07: Phase Results and Remaining Verifications

Phase statistics in the sharing material show the average cycle from requirement creation to completion dropped from 50.9 days to 13.8 days, a reduction of about 72.9%; window throughput rose from 63.9% to 98.9%.

Figure 8: Phase results and subsequent verification directions.
Figure 8: Phase results and subsequent verification directions.

Next verification items are clear: whether the same mechanism can replicate from individual projects to more modules; whether practices dependent on individual initiative can be沉淀 into stable team capabilities; whether shared result standards can enter daily evaluation rather than being effective only during a specific project push.

For teams wanting to try similar practices, start with one real problem: clarify the customer outcome to deliver and the acceptance evidence, designate one continuously responsible person, run the full process, then examine where time was spent, which handoffs can be reduced, and which risks still need expert judgment.

Agents evolve continuously around task results; teams must also work together around the same customer outcome. This is the direction the team is advancing and continues to verify in the Agentic SOC practice.

This article is included in the open-source project Alibaba AI Agent Handbook (https://aliyun.github.io/ai-agent-handbook/cases/chapter-25/agentic-soc-organization.html) for a better reading experience.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

security operationsorganizational restructuringmulti-agent collaborationAgentic SOCdelivery cycle reductionhuman-agent boundariesrole fusionself-evolution loopSOCBenchunified evaluation
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.