How We Built a 24/7 Autonomous User‑Feedback Pipeline with Qoder CLI

Background

As the Qoder product family expanded, the volume of user feedback surged, exposing the inefficiencies of a fully manual pipeline: operators exported Excel files, cleaned and categorized data, then manually entered tickets; developers spent at least 30 minutes per issue analyzing logs, with longer times for complex problems.

Goal

Build a 7×24 hour, fully unattended feedback processing system that automates submission, classification, clustering, log analysis, and code repair, leaving only the final code‑review step for humans.

System Architecture

The new backend consists of four core modules arranged as a pipeline:

Problem Classification : Filters invalid entries, splits feedback into product suggestions and defect reports, and further categorizes defects by business domain, replacing manual entry.

Problem Clustering : Groups similar defects to avoid duplicate noise, focusing downstream analysis on unique issues.

Log Analysis : For each defect, the system automatically reads the codebase, extracts user action traces, pinpoints root causes, and suggests fixes, eliminating manual log inspection.

Automatic Repair : When confidence is high, the agent generates a patch, creates a Code Review, and awaits human approval.

Technical Implementation

All AI capabilities are powered by Qoder CLI . The CLI was chosen over raw model APIs because it provides built‑in process management, concurrency handling, and isolation, which are critical for 24‑hour Harness Engineering workloads.

Installation in a Dockerfile: RUN curl -fsSL https://qoder.com/install | bash The personal access token is set via the QODER_PERSONAL_ACCESS_TOKEN environment variable, enabling subprocess calls to Qoder CLI.

Key CLI flags used: --yolo: Auto‑confirm mode, no interactive prompts. --model: Select model tier (Effective, Performance, Ultimate) – cheaper models for simple tasks, higher‑tier models for deep reasoning. --output-format=json: Structured JSON output for downstream parsing. --worktree: Isolated work area to avoid file‑write conflicts. --max-turns: Caps the number of LLM interaction rounds to prevent token waste.

Problem Classification Workflow

Filter out feedback lacking concrete information.

Separate remaining items into product suggestions vs. defect reports.

Determine whether a defect is valid.

Assign valid defects to a specific business sub‑category.

Because the task is simple, the --model Effective tier suffices, saving costs.

Problem Clustering Workflow

After classification, the CLI produces structured output, then runs a similarity‑matching round to cluster issues. Multimodal LLM understanding (text + screenshots) is used instead of traditional text similarity because identical problems can be described very differently.

A dynamic time window discards stale clusters, and a similarity threshold can be tuned in real time. A “inspection” agent can sample clusters and suggest threshold adjustments.

Log Analysis & Root‑Cause Identification

The agent leverages Qoder CLI’s tool‑calling abilities: it uses grep to search large logs efficiently, and may invoke web_search to check if similar issues exist in VS Code forums. Before finishing, the agent performs a self‑review: could fewer tool calls have found the root cause? Which steps were redundant?

The review is saved to task-retro.md and fed to a separate Pipeline Agent that updates the relevant Skill definitions.

Automatic Repair Workflow

The AI evaluates repair confidence and outputs a confidence score. A dynamic threshold triggers automatic patch generation. The --worktree flag allows concurrent repairs of different issues in the same repository.

Generated patches are submitted via the ticket system’s API, creating a Code Review for human sign‑off.

Cost‑Control Strategies

Two mechanisms limit credit consumption:

qodercli -p "..." --max-turns 80   # limit interaction rounds</code><code>timeout 1800 qodercli -p "..." --yolo   # enforce a 30‑minute wall‑clock limit

Empirical findings show that cheap models waste more tokens on complex tasks, while strong models finish deep‑reasoning tasks in a single pass, ultimately saving cost.

Model Tier Recommendations

Complex tasks (root‑cause analysis, code repair) : Use high‑tier models (Performance or Ultimate).

Simple tasks (clustering, trajectory extraction) : Use low‑cost models (Effective).

Results

Before automation, a single issue required ≥30 minutes of manual work. After deployment, the system completes root‑cause analysis in 2 minutes and runs continuously 24/7, with humans only reviewing final patches.

Continuous operation creates a self‑improving loop: each task’s retro‑analysis refines Skills, improving future performance.

Images