Tagged articles
pre-training
23 articles · Page 1 of 1
Experts at the Triple T summit debate whether embodied AI pre-training should prioritize massive human video datasets or real-robot interaction data, discussing tactile sensing, data recipes, evaluation methods, and cost reduction strategies for scaling embodied intelligence.
Multimodal DataScaling Lawscontinuous learning
0 likes · 14 min read

Baidu open-sources ERNIE-5.5-Flash-Base, a 280B-parameter sparse MoE model activating only 14B parameters per token, trained on 33.5T tokens with 128K context, achieving same-size SOTA against DeepSeek-V4-Flash-Base and others through hybrid sparse attention, multi-router latent MoE, and ECHO future-token prediction.
BaiduECHOERNIE-5.5-Flash-Base
0 likes · 19 min read

Google DeepMind released three new Gemini models—3.6 Flash with 65% token reduction, 3.5 Flash-Lite outperforming its larger sibling at lower cost, and 3.5 Flash Cyber for vulnerability detection—while confirming the start of pre-training for Gemini 4, signaling continued rapid advancement in production-ready AI agents.
AI agentsGeminiGoogle DeepMind
0 likes · 8 min read

Google DeepMind released three new Gemini models—3.6 Flash with 65% token reduction, 3.5 Flash-Lite outperforming larger models at lower cost, and 3.5 Flash Cyber for vulnerability detection—while confirming the start of 'most aggressive' pre-training for Gemini 4, signaling cheaper, faster AI agents for production.
AI agentsGeminiGoogle DeepMind
0 likes · 8 min read

Google DeepMind released three Gemini models—3.6 Flash (65% token savings on coding benchmarks), 3.5 Flash-Lite (350 tokens/sec, cheapest, beats larger model), and 3.5 Flash Cyber (security-focused, SOTA on CyberGym)—while confirming the most aggressive pre-training yet for Gemini 4, signaling cheaper, faster production AI agents.
AI agentsGeminiGoogle DeepMind
0 likes · 8 min read

Google DeepMind launched three new Gemini models—3.6 Flash with 65% token reduction, 3.5 Flash-Lite for high-speed low-cost processing, and 3.5 Flash Cyber for vulnerability detection—while confirming Gemini 4 pre-training has started.
AI agentsGeminiGoogle DeepMind
0 likes · 8 min read

Google DeepMind releases Gemini 3.6 Flash, 3.5 Flash-Lite, and 3.5 Flash Cyber with significant token savings, speed gains, and cybersecurity focus, while confirming aggressive pre-training for Gemini 4.
AI agentsBenchmarksGemini
0 likes · 8 min read

Google DeepMind launches Gemini 3.6 Flash, 3.5 Flash-Lite, and 3.5 Flash Cyber with lower token usage, cheaper pricing, and specialized capabilities, while confirming the most aggressive pre-training ever for Gemini 4.
AI agentsGeminiGoogle DeepMind
0 likes · 7 min read

Google DeepMind releases three specialized Gemini models — 3.6 Flash for token-efficient reasoning, 3.5 Flash-Lite for high-speed low-cost volume tasks, and 3.5 Flash Cyber for vulnerability detection — while confirming aggressive pre-training for Gemini 4, signaling a push to make production AI agents faster, cheaper, and more capable.
AI agentsGeminiGoogle DeepMind
0 likes · 7 min read

Google DeepMind released three new Gemini models—3.6 Flash with 65% token reduction, 3.5 Flash-Lite for high-speed low-cost processing, and 3.5 Flash Cyber for vulnerability detection—while simultaneously beginning aggressive pre-training for Gemini 4, signaling continued rapid advancement in AI agent capabilities and cost reduction.
AI agentsGeminiGoogle DeepMind
0 likes · 7 min read

The UC Berkeley 2025 PhD thesis by Linyuan Gong demonstrates that large language models for code must learn program structure—legal positions, complete units, and semantic constraints—through a series of code‑structure‑aware techniques, improving translation, completion, repair, and generation tasks.
ASTcode completioncode large language models
0 likes · 18 min read

A recent study shows that pre‑training Transformers on synthetic, non‑language data generated by Neural Cellular Automata can boost language‑model performance by up to 6%, accelerate convergence by 40%, and improve downstream reasoning, even outperforming models trained on massive natural‑text corpora.
Transformerattention mechanismsin-context learning
0 likes · 12 min read

The article explains how pre‑training gives LLMs general language abilities, how fine‑tuning and alignment turn them into usable assistants, why this creates a surge of applications across domains, and what inherent limitations and engineering safeguards are needed.
AI applicationsFine-tuningPrompt Engineering
0 likes · 8 min read

This article explains the three evolutionary stages of AI—from large language models that generate text, through workflow‑enhanced systems using retrieval‑augmented generation, to fully autonomous agents capable of self‑directed decision‑making—while detailing the four core technologies that power each stage.
AI evolutionAgentEmbedding
0 likes · 9 min read

The article breaks down the three core stages of large‑language‑model training—pre‑training, supervised fine‑tuning, and RLHF—explaining their purpose, methods, and concrete examples while noting DeepSeek‑R1’s recent breakthrough and its implications for AI development.
AI TrainingDeepSeekLLM
0 likes · 5 min read


ArchitectFeb 19, 2025 · Artificial Intelligence
The article critically examines whether the pre‑training Scaling Law still applies to Grok 3, compares its compute usage and model size with DeepSeek and OpenAI models, evaluates the cost‑effectiveness of pre‑training, RL and test‑time scaling, and explores how these insights shape future large‑language‑model development strategies.
Grok 3Model EfficiencyRL scaling
0 likes · 11 min read

The article examines Ilya Sutskever’s claim that pre‑training will end, argues that scaling laws still hold and data is not yet a bottleneck, highlights the scarcity of high‑quality frontier data, and explains why the industry is shifting toward inference‑time compute (o1) as a more sustainable path for large language models.
AI TrendsInference‑time ComputeScaling Laws
0 likes · 13 min read

This article provides a comprehensive overview of how large language models (LLMs) are applied in recommendation systems, covering two main paradigms—LLM+RS as a component and LLM as a standalone recommender—detailing their impact on pre‑training, fine‑tuning, prompting, and future research challenges.
Fine-tuningFuture DirectionsLLM
0 likes · 6 min read

This article examines the conditions under which graph neural network pre‑training is beneficial, proposes a data‑centric generator framework to assess transferability, introduces a data‑active pre‑training strategy that selects informative graphs, and presents experimental results showing that using less, well‑chosen data can outperform full‑scale pre‑training.
Graph Neural Networksdata selectiongraph generator
0 likes · 16 min read

This article provides a detailed overview of large language models (LLMs), tracing their evolution from statistical and neural language models to modern pre‑trained transformers, discussing scaling, training, adaptation, utilization, evaluation methods, available resources, and outlining current challenges and future research directions.
Model ScalingPrompt Engineeringevaluation
0 likes · 26 min read

Tencent Robotics X unveiled how its robot dog Max combines pre‑trained AI models with reinforcement learning across three learning stages, enabling it to acquire, store, and apply skills for autonomous decision‑making in complex tasks such as the World Chase Tag competition.
AIRoboticsSimulation
0 likes · 6 min read

The article introduces WuDaoMM and WuDaoCorpora 2.0, massive Chinese‑English multimodal datasets—including 650 million image‑text pairs, 3 TB of text, 93 TB of images, and 181 GB of dialogue—detailing their composition, formats, access options, and potential research applications.
Chinese AIMultimodal DatasetWuDaoMM
0 likes · 6 min read

This article provides an in‑depth overview of BERT, covering its NLP background, GLUE benchmark achievements, Transformer‑based architecture, pre‑training strategies (MLM and NSP), downstream fine‑tuning methods, and includes detailed PyTorch code implementations of its core components.
BERTNLPPyTorch
0 likes · 19 min read
