ThinkInAI Weekly #60: AI Shifts from Answers to Continuous Action Systems
This week's AI developments show a clear shift from model answer quality to continuous action capabilities, with Gemini 3.8 Live enabling concurrent voice-tool reasoning, Qwen3.8-Omni-Flash integrating long video understanding into agent workflows, and new frameworks like Salesforce Koa and Anthropic's metrics turning business rules and R&D automation into measurable systems.
Weekly Theme: From Model Answers to Runnable Systems
The central trend this week is a shift in AI product competition from "what the model can answer" to "how the model continuously gets things done." Real-time voice now combines dialogue, vision, reasoning, and tool calls in a single concurrent pipeline; full-modality models process hour-long audio/video and hand understanding results to tools; enterprises and research teams embed business rules, experimental data, shared memory, and supervision metrics into agent systems. Model capability remains important, but usability increasingly depends on the runtime system surrounding the model.
Headline 1: Gemini 3.8 Live — Conversation Without Interruption, Tools and Reasoning Run in Background
Google released Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking on September 15. The standard version targets high-concurrency, low-latency real-time dialogue; Extended Thinking reserves more compute for complex, multi-step tasks. Both accept audio, image, video, and text, and output speech and text, with a maximum context of 128K tokens. Access is via Gemini API and Google AI Studio; some enterprise channels remain in private preview.
Key Technical Shift
The most significant change is not "more human-like voice" but the transformation of the session into a concurrent system. Traditional voice assistants had to finish listening, transcribe, wait for the model, call tools, then synthesize speech. The new Live models can continue speaking while executing functions or API calls in the background. Extended Thinking performs multi-step reasoning while maintaining natural dialogue, using brief verbal acknowledgments and progress updates to signal task advancement. Visual context also enters the session in real time — for example, letting the model troubleshoot a device seen through the camera.
Technical Mechanisms
Native speech-to-speech path: Avoids fully serializing ASR, text LLM, and TTS into three separate modules.
Asynchronous function calling: Tool execution proceeds in parallel with audio output.
Incremental content updates: Structured data returned from tools is gradually merged into the ongoing conversation.
Multilingual support: Automatic detection and switching across 97 languages.
SynthID watermarking: Imperceptible AI-content watermark added to generated audio.
Engineering Trade-offs
Concurrency makes waiting feel natural but shifts complexity to application developers. Clients can no longer treat a single "turn end" as the model being completely idle; they must continuously handle audio frames, tool requests, tool results, cancellation signals, and background reasoning state. Tool calls must be designed for idempotency (repeat execution causes no extra side effects), permission boundaries, and failure recovery — otherwise a single natural-language utterance could trigger duplicate charges, duplicate orders, or state conflicts. Extended Thinking's complex-task ability also brings higher cost and longer tail latency, so customer-service, coaching, and field-assistant use cases need dynamic model-tier selection per task.
🔗 Google Official Release · Developer Notes · Model Card
Headline 2: Qwen3.8-Omni-Flash — Long Audio/Video Understanding Plugged into Agent Workflows
Alibaba Cloud launched Qwen3.8-Omni-Flash on September 17 and published more complete technical documentation over the weekend. The model accepts text, image, audio, and video, outputs text, provides 1M-token context, function calling, web search, implicit caching, and session caching. Its goal is not prettier multimodal QA but connecting long audio/video understanding to a complete pipeline of planning, tool execution, and result delivery.
Officials also expanded Qwen-MM-Plugins and opened Qwen-Live Harness. The former provides on-demand perception and tool capabilities for long media; the latter manages session execution under continuous real-time input. Typical tasks include generating structured notes from tutorial videos, extracting action items from long meetings, drafting narration for films, or having the model identify key segments and call editing tools to complete clipping. The model defaults to thinking mode; developers can adjust reasoning effort to trade off speed, cost, and complex-task capability.
Technical Mechanisms
Unified context across modalities: Text instructions, frames, sound, and video timelines jointly participate in reasoning.
Cache reuse: Already-processed audio/video context can be reused, reducing redundant overhead in multi-turn analysis.
Function calling + web search: Connects "understanding material" to external actions.
Qwen-MM-Plugins: Engineering interfaces for reading local segments, locating timestamps, and invoking specialized tools.
Engineering Trade-offs
Text-only output: The model does not directly generate speech; voice replies or speaker-preserving output still require combining other speech models or dubbing services.
1M context ≠ free long video: Sampling frame rate, image resolution, audio tokens, upload time, and caching strategy all affect billing and latency.
Privacy-sensitive content: Meeting, education, and film materials require clarity on cloud upload, regional services, and retention periods.
External tool reliability: End-to-end workflow reliability depends on whether external editing tools can execute model-specified time segments and operations via deterministic interfaces.
🔗 Model Documentation · Full-Modality Docs · Qwen-MM-Plugins
Selected 3: Salesforce Koa — Business Process Definitions Become Training Signals
Salesforce and NVIDIA released Koa on September 15, an enterprise reasoning model for CRM tool calling. Koa builds on NVIDIA Nemotron 3 Super, post-trained on public data and synthetic enterprise scenarios. Salesforce states no customer data was used; model weights, post-training, and inference stay within their trust boundary; currently available only to select Agentforce customers in pilot.
Technical Mechanisms
The core is Agent Script : the same business specification that describes routing, sub-agents, typed actions, tool permissions, and termination conditions is also compiled into a training environment. The system expands workflows into multi-turn tasks with roles and scenarios, then trains the model using supervised fine-tuning and Group Relative Policy Optimization (GRPO — a method that updates policy by comparing multiple trajectories on the same problem). Synthetic users, tool simulators, and coverage evaluators jointly judge task completion; repeated tool calls receive explicit penalties.
Engineering Trade-offs
Specification-driven training suits enterprise tasks with clear processes, fixed tools, and explicit compliance requirements, but the model inherits biases from the simulation environment.
Salesforce's internal CRM benchmark shows Koa makes fewer errors, but this remains a vendor-internal evaluation and cannot be generalized to all enterprise work.
120B-parameter deployment cost is significant; "based on open weights" does not mean Koa itself is open-weight.
🔗 Official Announcement · Technical Documentation · Paper
Selected 4: Periodic Neon — Lab Data Enters the Scientific Agent Loop
Periodic Labs released Neon on September 15: a 1-trillion-parameter scientific model continued from the Kimi K2.6 open-weight model, for analyzing powder X-ray diffraction (XRD — determining material composition from crystal scattering patterns). The model is deployed in their materials lab for high-throughput superconductor and magnet research. Officials report 55.3% success on 134 difficult internal samples versus 2.7% for the base model; these numbers come from Periodic's own environment and still need external replication.
Technical Mechanisms
Scientific mid-training: Papers, code, and experimental data.
Long-context, multimodal RL: Lab data used for reinforcement learning.
Expert scoring rules + LLM judges: XRD results are hard to auto-grade; material experts defined detailed scoring rules, and multiple LLMs formed a judge panel to provide training and evaluation signals.
Scientific harness: Manages tools, context, and execution loops; integrates experimental conditions, internal structure databases, simulation data, and analysis software.
Async generation/training infrastructure: Generation and training run asynchronously; model-generated code executes in pbox sandboxes on idle CPUs of GPU nodes.
Engineering Trade-offs
55.3% success means over 40% of difficult samples still fail.
Internal data, tools, and benchmarks make the system close to real experiments but reduce external reproducibility.
LLM judge–expert agreement is not 100%.
Physical-world limits: Lab equipment cannot be infinitely replicated like digital environments; a single experiment may take days, and results can be ambiguous. Scaling model compute does not automatically eliminate experimental throughput and label uncertainty.
🔗 Official Announcement · Research Documentation · Infrastructure Documentation
Selected 5: Claude Code Projects — One Orchestrator Manages Multiple Cloud Threads
On September 17, Anthropic turned Claude Code Projects from a "project folder" into a persistent conversation. After the user states a goal, an orchestrator splits scope, dispatches tasks, coordinates parallel threads, checks outputs, and aggregates results. Each thread is an independent Claude Code cloud session with its own repository copy and code branch; threads share project memory and a knowledge base, and can continue to invoke sub-agents, loops, or workflows.
Technical Mechanisms
Orchestrator retains long-term goals, decisions, and progress.
Worker threads handle concrete tasks only.
Separation of concerns: "Who remembers context" and "who executes edits" are decoupled. The orchestrator creates or reuses threads per task; threads complete tests, edits, and pull requests, then report results back.
Shared memory lets new threads inherit release dates, architectural decisions, sensitive directories, etc., reducing repeated prompting.
Engineering Trade-offs
Merge conflicts when parallel threads edit the same code must be resolved like ordinary pull requests.
Shared memory can propagate erroneous judgments; critical constraints should remain in reviewable repository files and tests.
Each thread is a full session , consuming usage quota faster.
Cloud-only launch: Local tools, on-premise code, and local environment support not yet available; initially limited to select Pro and Max users.
🔗 Official Release
Selected 6: Anthropic R&D Automation Metrics — Breaking "AI Helps AI" into Measurable Questions
On September 17, Anthropic published three internal metric sets: how much R&D work Claude leads, how research-agent actions are supervised, and how much compute serves safety. As of August, Claude was rated "leading" on 26% of R&D work, over 90% reached "collaborative" or higher, but no measured subset achieved full autonomy. Their largest internal platform runs ~30,000 research and engineering agents concurrently.
Technical Mechanisms
Task tree: From July work logs, ~15,000 fine-grained tasks were organized into a fixed tree of 542 nodes.
Automation levels AL0–AL5: Each task category evaluated against this scale.
Agent supervision decomposed into: coverage, review latency, escalation rate. Online monitoring checks before action execution; offline monitoring reads full activity afterward.
Compute allocation: Combines capacity-planning tags with cloud service reports, then classifies which work primarily serves safety.
Engineering Trade-offs
Single-company, self-reported, short-window snapshot.
Evaluation uses Anthropic's own models, potentially sharing biases with the system under test.
"100% of actions monitored" does not mean monitoring catches 100% of anomalies.
R&D task tree evolves with the organization; safety vs. capability research boundaries are fuzzy. This metric suite is better for tracking one organization over time, not for cross-lab ranking.
🔗 Official Measurement Report
Selected 7: HarnessTax — Same Model, Different Runtime Framework Can Yield 5× Cost Difference
UC Berkeley and the Arena team released HarnessTax on September 16, comparing seven models across three coding-agent frameworks: Claude Code, Codex CLI, and Pi. The study formed 21 "model + framework" combinations, sampled 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, and ran each combination three times per task. They report that under this experimental setup, the same model across frameworks yields similar success rates, but cost can differ by up to ~5×.
Technical Mechanisms
The harness (runtime framework) determines system prompts, tool definitions, file-editing methods, context retention, execution loops, caching, and stop conditions. Even with the same base model, how much instruction the framework sends on the first call, and how much tool schema it repeats each round, changes token consumption. Findings include:
Claude Code's average initial context exceeds Pi's by >10×.
Pi provides only four tools ( read, write, edit, bash) yet still reaches the cost–success-rate Pareto frontier on both benchmarks.
Engineering Trade-offs
Results cover only two public benchmarks and 30-task samples; confidence intervals are wide.
Models may have seen related tasks during training.
"Round" definitions differ across frameworks.
Some networking capabilities were disabled.
Subscription pricing, human review, long-term maintenance of real codebases, interaction experience, and proprietary features were not included. Therefore "simpler framework is cheaper" cannot directly equal "all teams should switch frameworks."
🔗 Research Project Page · Official Repository
Additional Weekly Updates (Brief)
Jev (TypeSafe AI): Returns probabilities over predefined choices/ratings/booleans instead of free text; "no hallucination" means output stays within preset types, not that judgments are always correct; 70–500 ms latency per vendor self-test.
Devin Code Scans: MapReduce-style agentic scan across entire codebases; may miss cross-shard call chains; large auto-findings increase review queue and quota cost.
Claude Code 2.1.277: Falls back to AGENTS.md when CLAUDE.md is absent; conditional fallback, not a merge; teams with both files still need a single source of truth; not yet supported on Bedrock, Vertex AI, or Foundry.
Cowork merged into Claude: Long-running multi-step work folded into regular conversations; projects, connectors, skills, artifacts retained; Docs and Slides enter paid-plan testing.
Qwen3.8-LiveTranslate: Thinker–Talker two-stage design; interleaves video, audio, source text, translation; preserves speaker timbre; 60 input languages, 29 with speech output; ~2.3 s avg latency; no function calling, web search, or batch; production must test noise, cross-talk, network jitter.
Bonsai 2 27B: Ternary weights (−1, 0, +1) with scaling factors; 27B model ~5.95 GB PTQ1_0; requires matching llama.cpp runtime; throughput, memory, and task quality need joint testing on target device; 98.2% performance retention per vendor summary.
Needle 3: Layer-sliceable network; one weight set deploys 2–20 layers, ~8–29 MB; targets on-device tool calling and structured extraction; sacrifices general chat for small size and deterministic output; large tool-catalog retrieval, confidence calibration, complex reasoning still limited.
Laya: ModernBERT/mmBERT encoder outputs choice/rating/boolean probabilities with temperature-scaled calibration; English, multilingual, and specialized fine-tunes released; base model near random on specific decision benchmarks; high-cardinality options limited by output-head budget; suited for self-hosted experiments, not drop-in replacement for mature classifiers.
Nimble (Bespoke-Nimble-9B): LoRA on Qwen3.5-9B; reads raw candidate scores, code assembles typed probability outputs; open model, data method, code; text-only, flat schema, ≤26 string options; 324 synthetic holdout set not representative of broad production tasks.
Occamy: Paper published September 4; this week only code repo release. Post-trained from Qwen3.6-35B-A3B; long-horizon execution and broad agent abilities trained separately then merged; multiple quantized weights released; retrieval-heavy tasks and native desktop vision interaction remain boundaries.
ZGCM-1: 7.39B dense model trained from scratch; interleaved sliding-window and global attention; context extended to 256K; code and paper released September 10/11; requires trust_remote_code=True — review custom repo code before deploy.
Meridian: Agent Skills continue earlier marketing-measurement updates; skills generate modeling, diagnostics, visualization, budget-optimization scripts; does not eliminate data quality, causal assumptions, Bayesian priors, or MCMC sampling cost; human checkpoints still required.
R2T2 (Confucius4-R2T2, NetEase Youdao): Streaming ASR that only appends the longest stable prefix; avoids downstream agent state thrashing from transcription rollback; supports 80 ms–2 s audio chunks; smaller chunks = faster response but less context, higher error rate; code and weights under different licenses.
JING & DAO: JING generates what a character sees/hears now; DAO maintains shared world beyond the camera; observer filter passes local state to JING; public artifacts focus on JING, DAO still prototype; fast motion, long-session spatial drift, latency, cost far from economic real-time deployment.
Dream-RSI: Builds replay simulator from historical coding-agent discovery trees; lets orchestration layer compare exploration strategies without repeated expensive real evaluations; improves search/orchestration strategy, not model weights; historical trajectory bias can overfit simulator; still needs real eval validation.
MiniMax Code: Product existed earlier; this week terminal coding agent repo open-sourced (MIT); supports project reading, file editing, test running, interactive UI, headless execution, multiple compatible model interfaces; install script writes to user directory and PATH; hosted tool integration involves networking, account permissions, balance.
Weekly Observation: Agents Enter "System Engineering Week"
The strongest common trend is not a model gaining more parameters, but models being placed into increasingly complete systems. Real-time voice needs async state machines; full-modality agents need media caches and toolchains; enterprise models need business specs that generate training environments; scientific models need lab data and specialized software; multi-agent programming needs orchestrators, branches, and shared memory. HarnessTax reminds us: these outer systems directly change success rates, costs, and maintainability.
For product teams, the next step is not just asking "which model is best" but recording four numbers on their real tasks: completion rate, unit completion cost, human review time, and failure recovery rate. Only by putting model, runtime framework, tool permissions, and evaluation data on the same spreadsheet can you see whether an agent system is truly production-ready.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
