Tagged articles

cross-entropy

11 articles · Page 1 of 1

Oct 25, 2026 · Artificial Intelligence

Google Discovers Two 20-Year-Old Bugs in Softmax-Cross-Entropy Implementation

Google researchers identify two numerical bugs in the standard softmax-cross-entropy implementation used for two decades: Softmax Collapse zeros out gradients beyond a precision cliff, and Softmax Cancellation creates a noise floor in distillation; fixes using stable log-space computations eliminate Slingshot loss spikes in grokking and improve convergence across precisions.

Adam optimizerSlingshotcross-entropy

0 likes · 8 min read

Google Discovers Two 20-Year-Old Bugs in Softmax-Cross-Entropy Implementation

Tencent Technical Engineering

Oct 13, 2026 · Artificial Intelligence

How LLMs Work: From Next-Token Prediction to Intelligence

This comprehensive guide explains large language models from first principles, covering token prediction, information theory, neural networks, Transformer architecture, training via gradient descent, and the path from base models to chat assistants like ChatGPT, including limitations such as hallucinations.

Attention MechanismBackpropagationGradient Descent

0 likes · 74 min read

How LLMs Work: From Next-Token Prediction to Intelligence

Machine Heart

Sep 9, 2026 · Artificial Intelligence

Action Map Policy: Pixel Classification for High-Precision Robot Manipulation

Action Map Policy (AMP) reframes robot manipulation as pixel-level classification by projecting 3D keypoint trajectories onto 2D image planes, enabling cross-entropy loss to model multi-modal action distributions with sub-millimeter precision and single-forward-pass inference.

Action Map PolicyImitation LearningMulti-View Transformer

0 likes · 12 min read

Action Map Policy: Pixel Classification for High-Precision Robot Manipulation

Baobao Algorithm Notes

Oct 28, 2025 · Artificial Intelligence

Why Entropy Collapse Limits LLM Reinforcement Learning and How to Fix It

The article explains how information entropy, cross‑entropy, and KL‑divergence shape reinforcement learning for large language models, describes the phenomenon of entropy collapse, compares token‑level and policy‑level entropy, and reviews recent methods like Clip‑Cov and KL‑Cov that mitigate this issue.

cross-entropyentropypolicy entropy

0 likes · 11 min read

Why Entropy Collapse Limits LLM Reinforcement Learning and How to Fix It

Data STUDIO

Sep 15, 2025 · Artificial Intelligence

Understanding Linear and Logistic Regression: From MSE to Cross‑Entropy

The article explains linear regression and logistic regression fundamentals, covering loss functions such as mean‑squared error and cross‑entropy, analytic solutions, feature expansion for non‑linear separability, and provides Python code examples to illustrate the concepts.

Machine LearningPythoncross-entropy

0 likes · 7 min read

Understanding Linear and Logistic Regression: From MSE to Cross‑Entropy

Model Perspective

Sep 10, 2024 · Artificial Intelligence

Why Cross-Entropy Is the Key Loss Function for Classification Models

This article explains how loss functions evaluate model performance, contrasts regression’s mean squared error with classification’s cross‑entropy, describes one‑hot encoding and softmax outputs, and shows why higher predicted probabilities for the correct class yield lower loss, highlighting applications in image, language, and speech tasks.

ClassificationMachine LearningOne-hot encoding

0 likes · 5 min read

Why Cross-Entropy Is the Key Loss Function for Classification Models

Rare Earth Juejin Tech Community

Nov 16, 2022 · Artificial Intelligence

Deep Learning Semantic Segmentation: FCN Source Code Analysis

This tutorial walks through the complete FCN pipeline for semantic segmentation, covering VOC dataset loading, data augmentation, collate functions, model construction, training loops, loss computation with cross‑entropy (including ignore‑index handling), and inference, while providing full PyTorch code snippets for each step.

FCNPyTorchVOC dataset

0 likes · 19 min read

Deep Learning Semantic Segmentation: FCN Source Code Analysis

Code DAO

May 16, 2022 · Artificial Intelligence

How to Build a Simple Neural Network from Scratch with NumPy

This article walks through implementing a basic multi‑layer neural network using only NumPy, covering terminology, network architecture, forward and backward propagation, activation functions, loss calculation, parameter updates with SGD, and compares the custom model with a Keras implementation.

BackpropagationMachine LearningNeural Network

0 likes · 17 min read

How to Build a Simple Neural Network from Scratch with NumPy

Code DAO

May 6, 2022 · Fundamentals

Information Theory Foundations for Machine Learning and Deep Learning

The article explains Shannon information content, entropy, cross‑entropy, KL‑divergence, conditional entropy and mutual information, illustrating each concept with coin‑flip and dice examples, visual formulas, and discusses their roles as loss functions and evaluation metrics in machine‑learning models.

Information TheoryKL DivergenceMachine Learning

0 likes · 8 min read

Information Theory Foundations for Machine Learning and Deep Learning

21CTO

Feb 7, 2018 · Artificial Intelligence

Demystifying Entropy: From Basic Concepts to Cross‑Entropy and KL Divergence

This article explains entropy, joint entropy, conditional entropy, and related measures such as KL divergence and cross‑entropy, using intuitive coin‑flip examples and mathematical formulas to show how they quantify uncertainty and information in probability distributions.

Information TheoryKL DivergenceMachine Learning

0 likes · 14 min read

Demystifying Entropy: From Basic Concepts to Cross‑Entropy and KL Divergence

Architecture Digest

Feb 3, 2018 · Artificial Intelligence

Understanding Entropy, Joint Entropy, Conditional Entropy, Relative Entropy, and Cross Entropy

This article explains the concepts of entropy, joint entropy, conditional entropy, relative entropy (KL divergence) and cross‑entropy, illustrating their definitions, mathematical formulas, intuitive interpretations, and relationships through simple probability examples and visual diagrams.

Information TheoryKL DivergenceMachine Learning

0 likes · 14 min read

Understanding Entropy, Joint Entropy, Conditional Entropy, Relative Entropy, and Cross Entropy