BestHub
Sign in
Tagged articles

softmax

8 articles · Page 1 of 1
PaperAgent
PaperAgent
Oct 25, 2026 · Artificial Intelligence

Google Discovers Two 20-Year-Old Bugs in Softmax-Cross-Entropy Implementation

Google researchers identify two numerical bugs in the standard softmax-cross-entropy implementation used for two decades: Softmax Collapse zeros out gradients beyond a precision cliff, and Softmax Cancellation creates a noise floor in distillation; fixes using stable log-space computations eliminate Slingshot loss spikes in grokking and improve convergence across precisions.

Adam optimizerSlingshotcross-entropy
0 likes · 8 min read
Google Discovers Two 20-Year-Old Bugs in Softmax-Cross-Entropy Implementation
Lisa Notes
Lisa Notes
Jun 23, 2026 · Artificial Intelligence

Understanding NLP Activation Functions: The Role of Softmax

The article explains how the softmax activation function converts neural network outputs into probability distributions for multi‑class NLP tasks, describes its mathematical form and S‑shaped behavior, and discusses the inductive approach, data quality, training objectives, and interpretability challenges in deep learning language models.

NLPactivation functiondata quality
0 likes · 4 min read
Understanding NLP Activation Functions: The Role of Softmax
Code Archaeology
Code Archaeology
Sep 19, 2025 · Artificial Intelligence

Neural Networks Explained: Why ReLU Beats Sigmoid & How to Build One in PyTorch

This article explains neural network fundamentals, covering why logistic regression units form neurons, the problems with Sigmoid activation in hidden layers (vanishing gradients, non-zero centered outputs), why ReLU and its variants are preferred, how Softmax enables multi-class classification with cross-entropy loss, and provides a complete PyTorch implementation example.

Neural NetworksPyTorchReLU
0 likes · 8 min read
Neural Networks Explained: Why ReLU Beats Sigmoid & How to Build One in PyTorch
AI Algorithm Path
AI Algorithm Path
Feb 19, 2025 · Artificial Intelligence

How Temperature Shapes Output in Large Language Models

The article explains the Temperature hyper‑parameter in large language models, shows how it modifies the softmax distribution, provides a Python visualisation script, and demonstrates through experiments that higher values increase creativity while lower values make outputs more deterministic.

Large Language ModelsPythonsampling
0 likes · 5 min read
How Temperature Shapes Output in Large Language Models
Model Perspective
Model Perspective
Sep 10, 2024 · Artificial Intelligence

Why Cross-Entropy Is the Key Loss Function for Classification Models

This article explains how loss functions evaluate model performance, contrasts regression’s mean squared error with classification’s cross‑entropy, describes one‑hot encoding and softmax outputs, and shows why higher predicted probabilities for the correct class yield lower loss, highlighting applications in image, language, and speech tasks.

ClassificationMachine LearningOne-hot encoding
0 likes · 5 min read
Why Cross-Entropy Is the Key Loss Function for Classification Models
dbaplus Community
dbaplus Community
Nov 10, 2016 · Artificial Intelligence

Demystifying Recurrent Neural Networks: Theory, Training, and Implementation

This article explains the fundamentals of recurrent neural networks (RNNs), their role in language modeling, various RNN architectures such as bidirectional and deep RNNs, the back‑propagation through time (BPTT) training algorithm, gradient challenges, vectorization techniques, and provides a step‑by‑step code implementation.

BPTTRNNRecurrent Neural Network
0 likes · 21 min read
Demystifying Recurrent Neural Networks: Theory, Training, and Implementation