ByteDance Interviewer: Agent Reasoning Slow & Costly? How to Optimize Efficiency
This article reviews the 'Toward Efficient Agents' survey paper, which systematically tackles AI agent efficiency through memory, tool learning, and planning optimizations—covering techniques like context compression, parallel tool calling, and fast/slow planning—while stressing holistic optimization across all three dimensions for practical, cost-effective deployment.
What Is Efficiency?
The paper opens with a clear definition: efficiency is a balance between two dimensions—maximizing effectiveness under fixed cost, and minimizing cost for a given level of effectiveness. The author notes that many agents are functionally powerful but take tens of seconds and several dollars per inference, making them impractical for production. This is framed as an engineering problem, not merely a technical one.
Three Core Components
The survey breaks agent efficiency optimization into three parts: memory, tool learning, and planning. The author relates each to personal experience building a marketing agent.
Efficient Memory
Memory is divided into working memory (internal to the model) and external memory. Working memory approaches include COMEDY, which compresses context via dialogue distillation, and MemoryLLM, which stores memories directly in latent space—fast access but limited capacity. External memory offers more flexibility: project-based storage, graph structures, or systems like MemoryBank that manage retention with a forgetting curve. The author's marketing agent used a vector database to store user history and preferences, retrieving via similarity search.
The paper identifies three memory management strategies:
Rule-based: fixed pruning rules, low cost but lacks adaptability.
LLM-based: dynamic decisions on what to add, delete, or modify, adaptive but resource-intensive.
Hybrid: rules trigger LLM optimization, balancing cost and effectiveness.
The author adopted the hybrid approach: rules filter out obviously unimportant chatter (e.g., greetings), then an LLM judges which information deserves long-term storage.
Tool Learning
Tool learning covers three stages: selection, invocation, and integrated reasoning.
Tool selection – quickly finding the best tool among hundreds or thousands. ProTIP uses contrastive learning for matching; TinyAgent employs a small model for filtering. The core idea is reducing the search space, akin to recommendation systems.
Tool invocation – strategy matters. ToolFormer embeds call parameters directly into context; LLMCompiler enables parallel invocation; BTP adds budget constraints. The author illustrates: serial calls to five tools might take 5 seconds, while parallel calls could finish in 1 second.
Integrated reasoning – the LLM decides whether a tool is needed at all, like an expert who first considers if they can answer directly without consulting a manual.
Efficient Planning
Planning is the most central and costly capability.
Single-agent planning – SwiftSage adopts a fast/slow thinking mode: simple tasks use "fast thinking" for direct answers; complex tasks use "slow thinking" with multi-step reasoning. This mirrors human cognition, where ~90% of decisions are intuitive and only ~10% require deep thought.
Multi-agent collaboration – Chain-of-Agent changes the interaction topology from all-to-all to linear passing, drastically cutting communication overhead. An even more aggressive method distills multi-agent capabilities into a single model, reducing inference cost by an order of magnitude.
Practical Value
The paper compiles benchmarks that evaluate both effectiveness and efficiency: MemBench for memory, ToolBench for tools, and TPS-Bench for planning. These measure token consumption, latency, and invocation counts alongside quality.
The author emphasizes that agent developers must not only ask "can it do the task?" but also "can it do it well, fast, and cheap?" End users care about problem resolution within acceptable time and cost, not raw intelligence. The key takeaway: efficiency optimization requires systematic thinking across memory, tools, and planning together.
Paper link: https://efficient-agents.github.io/
Repository: https://github.com/yxf203/Awesome-Efficient-Agents
Code example
{
"target"
:
"简单认识我"
,
"selfInfo"
:{
"genInfo"
:
"大厂面试官,中科院自所硕,从事数据闭环业务、RAG、Agent等,承担技术+平台的偏综合性角色。善于调研、总结和规划,善于统筹和协同,喜欢技术,喜欢阅读新技术和产品的文章与论文"
,
"contactInfo"
:
"abc061200x, v-adding disabled"
,
"slogan"
:
"简单、高效、做正确的事"
,
"extInfo"
:
"喜欢看电影、喜欢旅游、户外徒步、阅读和学习,不抽烟、不喝酒,无不良嗜好"
}
}Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
