BestHub
Sign in
Tagged articles

quantization

157 articles · Page 1 of 2
DataFunTalk
DataFunTalk
Aug 11, 2026 · Artificial Intelligence

Meta Open‑Sources Muse Glimmer: Packing a 30B Model into 24 GB VRAM for Always‑On Local Agents

Meta’s Muse Glimmer is a 30‑billion‑parameter, open‑source dense model engineered to run continuously on consumer‑grade hardware with as little as 24 GB VRAM, using 4‑bit quantization and DFlash speculative decoding to retain multimodal, long‑context and agentic capabilities while achieving up to 233 tok/s throughput.

Muse GlimmerOpen source AIagentic benchmarks
0 likes · 10 min read
Meta Open‑Sources Muse Glimmer: Packing a 30B Model into 24 GB VRAM for Always‑On Local Agents
Machine Heart
Machine Heart
Aug 10, 2026 · Artificial Intelligence

How MiniMax H3’s Open‑Source Week Is Rewriting the DeepSeek Story for Video AI

MiniMax H3’s open‑source release sparked a week of explosive community activity, with rapid ecosystem integration, aggressive quantization to fit consumer GPUs, speed‑up LoRAs, diverse creative workflows, and industry attention that together illustrate how a powerful video model can reshape the AI landscape.

AI communityComfyUIMiniMax-H3
0 likes · 18 min read
How MiniMax H3’s Open‑Source Week Is Rewriting the DeepSeek Story for Video AI
Data Party THU
Data Party THU
Aug 8, 2026 · Artificial Intelligence

Memory-Efficient Algorithms for Large Language Model Inference

The article reviews Coleman Hooper's 2026 Berkeley PhD thesis, which shows that LLM inference is increasingly limited by memory bandwidth and capacity, and proposes a four‑pronged approach—weight quantization, KV‑cache quantization, selective context loading, and multipole attention—to dramatically improve memory efficiency and throughput.

KV cacheLLMMemory Efficiency
0 likes · 12 min read
Memory-Efficient Algorithms for Large Language Model Inference
Old Zhang's AI Learning
Old Zhang's AI Learning
Aug 4, 2026 · Artificial Intelligence

The “Best” Qwen3.6-27B Variant: A God‑Level Model for Local Deployment

The community‑fine‑tuned Qwen3.6-27B‑Fable‑Fusion‑711 model combines multi‑stage fine‑tuning, model fusion and uncensored processing, delivers a 0.711 ARC‑C score that surpasses the original on six of seven benchmarks, and offers a rich set of GGUF quantizations with detailed performance guidance for local deployment.

AIGGUFQwen3.6-27B
0 likes · 10 min read
The “Best” Qwen3.6-27B Variant: A God‑Level Model for Local Deployment
DataFunSummit
DataFunSummit
Aug 1, 2026 · Artificial Intelligence

From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless

This article presents a detailed technical guide on constructing a multimodal product search system that combines text and image queries using embedding vectors, hybrid dense‑sparse models, vector similarity metrics, and Elasticsearch Serverless’s built‑in vector support, while also covering quantization techniques, hybrid search fusion, and best‑practice deployment architectures on Alibaba Cloud.

AI search platformElasticsearch ServerlessHybrid Search
0 likes · 22 min read
From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless
PaperAgent
PaperAgent
Aug 1, 2026 · Artificial Intelligence

How to Run DeepSeek‑V4‑Flash Locally on a 100 GB Server: Best‑Practice Guide

The article details the release of DeepSeek‑V4‑Flash‑0731, explains how its 284 B‑parameter, 13 B‑activated model can run losslessly on a machine with only 169 GB RAM using Unsloth’s UD‑Q8_K_XL quantization, compares quantization quality, and provides step‑by‑step deployment instructions via Unsloth Studio and llama.cpp.

AI Model DeploymentDeepSeek-V4-FlashLossless Quantization
0 likes · 7 min read
How to Run DeepSeek‑V4‑Flash Locally on a 100 GB Server: Best‑Practice Guide
AI Engineering
AI Engineering
Aug 1, 2026 · Artificial Intelligence

Running DeepSeek V4 Flash 284B Locally – Performance Beats V4 Pro

DeepSeek V4 Flash 0731, a 284‑billion‑parameter model with 13 B active weights and a 1 M context window, can run locally using Unsloth's lossless GGUF quantizations on machines with 128‑169 GB memory, and its benchmark scores surpass the V4 Pro preview.

AI AgentDeepSeekLocal Inference
0 likes · 5 min read
Running DeepSeek V4 Flash 284B Locally – Performance Beats V4 Pro
AI Programming Lab
AI Programming Lab
Jul 28, 2026 · Artificial Intelligence

How a Team Ran the Open‑Source Kimi K3 Model on 80 RTX 5090 GPUs

The Kimi K3 model weights were released on HuggingFace (1.56 TB total), featuring mixed attention, Attention Residuals, and a Stable LatentMoE that together cut scaling cost by 2.5×, and a detailed cost‑benefit analysis shows how 80 consumer‑grade RTX 5090 cards can run the full 2.8‑trillion‑parameter model with 20 tok/s throughput, while highlighting memory‑saving quantization, KV‑cache design, and the steep price gap versus professional GPUs.

AI Model DeploymentKimi K3RTX 5090
0 likes · 9 min read
How a Team Ran the Open‑Source Kimi K3 Model on 80 RTX 5090 GPUs
AI Programming Lab
AI Programming Lab
Jul 21, 2026 · Artificial Intelligence

How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture

The article examines how Chinese LLMs such as Kimi K3 (2.8 T) and Qwen 3.8‑Max (2.4 T) achieved 2‑trillion‑parameter scales by progressively decoupling parameters from compute with Mixture‑of‑Experts, optimizing attention, and introducing training tricks like Muon and low‑precision quantization, tracing four years of architectural advances.

Attention MechanismsLarge Language ModelsMixture of Experts
0 likes · 11 min read
How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture
MaGe Linux Operations
MaGe Linux Operations
Jul 16, 2026 · Artificial Intelligence

How to Choose Between INT8, FP8, and INT4 Quantization for Large Models

This guide explains how to evaluate INT8, FP8, and INT4 quantization strategies for large language models on NVIDIA GPUs, covering precision trade‑offs, memory consumption, kernel support, KV‑Cache considerations, and detailed deployment, testing, and rollback procedures to ensure performance and quality.

FP8GPU deploymentINT4
0 likes · 48 min read
How to Choose Between INT8, FP8, and INT4 Quantization for Large Models
Old Zhang's AI Learning
Old Zhang's AI Learning
Jul 8, 2026 · Artificial Intelligence

Deploying the Quantized DeepSeek‑V4‑Flash Locally: Performance, Benchmarks, and Tips

This article walks through DeepSeek‑V4‑Flash, a lightweight 284B MoE model with 13B active parameters and 1‑million‑token context, explains its three thinking modes, presents official benchmark results showing Flash can match Pro with higher reasoning budget, and provides step‑by‑step local deployment instructions using Unsloth’s GGUF quantization and llama.cpp fixes.

AIDeepSeek-V4-FlashGGUF
0 likes · 13 min read
Deploying the Quantized DeepSeek‑V4‑Flash Locally: Performance, Benchmarks, and Tips
AntData
AntData
Jul 8, 2026 · Databases

Deep Dive into SINDI: High‑Performance Sparse Vector Search Index for VSAG

The article explains SINDI, a sparse‑vector retrieval index designed for VSAG, detailing its window‑based weighted inverted lists, multi‑layer pruning, quantization and reranking techniques that achieve dramatically faster index construction, smaller index size, and high query throughput compared with traditional inverted or graph indexes.

Maximum Inner Product SearchPruningSparse Vector
0 likes · 21 min read
Deep Dive into SINDI: High‑Performance Sparse Vector Search Index for VSAG
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 6, 2026 · Artificial Intelligence

Running GLM‑5.2 on AMD: 2,626 Tokens/s at Half Nvidia’s Cost

Wafer demonstrates that the open‑source GLM‑5.2 model can run on AMD’s MI355X GPU at 2,626 tokens per second with single‑stream throughput of 213 tokens per second, achieving comparable performance to Nvidia’s B200 at less than half the cost, by applying MXFP4 quantization, selecting the sglang engine, and fixing naming and kernel issues.

AMD MI355XCostGLM-5.2
0 likes · 9 min read
Running GLM‑5.2 on AMD: 2,626 Tokens/s at Half Nvidia’s Cost
DataFunSummit
DataFunSummit
Jul 6, 2026 · Artificial Intelligence

From Text to Images: Building Multi‑Modal Product Search with Elasticsearch Serverless

The article explains how modern e‑commerce search is evolving from simple keyword matching to multi‑modal retrieval, outlines a generic architecture that combines text and image embeddings, describes vector similarity metrics and quantization techniques, and demonstrates how Elasticsearch Serverless and Alibaba Cloud AI Search Platform enable a low‑cost, fully managed end‑to‑end multi‑modal product search solution.

ElasticsearchEmbeddingMulti-modal Search
0 likes · 21 min read
From Text to Images: Building Multi‑Modal Product Search with Elasticsearch Serverless
ThinkingAgent
ThinkingAgent
Jul 5, 2026 · Artificial Intelligence

Building L1 AI Infra: Model Gateways, Smart Routing, and High‑Performance Inference Engines

The article presents a comprehensive, production‑ready guide for the L1 layer of AI infrastructure, detailing how model gateways unify calls, intelligent routing selects the optimal model, inference engines maximize GPU throughput, and quantization and KV‑Cache techniques dramatically cut costs while maintaining performance.

AI infrastructureKV cacheLLM routing
0 likes · 25 min read
Building L1 AI Infra: Model Gateways, Smart Routing, and High‑Performance Inference Engines
Black & White Path
Black & White Path
Jun 30, 2026 · Artificial Intelligence

A 27B Red‑Team AI Model That Runs on Just 12 GB VRAM

The BugTraceAI CORE Ultra 27B model, fine‑tuned on 2,541 real vulnerability reports, generates fully functional Nuclei templates, CVE PoCs, webshell bypasses, JWT cracking tools, and kernel exploits with a 0 % rejection rate, and its quantized Q4 version runs on a single 24 GB GPU, making advanced red‑team automation accessible.

BugTraceAIGPULLM
0 likes · 7 min read
A 27B Red‑Team AI Model That Runs on Just 12 GB VRAM
AI Tech Publishing
AI Tech Publishing
Jun 29, 2026 · Artificial Intelligence

Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output

The article walks through every stage of LLM inference—from tokenization and embedding, through transformer layers, prefill (compute‑bound) and decode (memory‑bound) phases, KV‑cache management, attention redesign, quantization, and serving infrastructure—explaining how each step consumes time and resources and how to identify bottlenecks.

Attention optimizationKV cacheLLM inference
0 likes · 14 min read
Understanding the Full LLM Inference Pipeline: From Tokenization to Streaming Output
Raymond Ops
Raymond Ops
Jun 27, 2026 · Artificial Intelligence

vLLM Quantized Inference: Loading AWQ/GPTQ Models and Optimizing GPU Memory

This article provides a step‑by‑step guide on using vLLM to load AWQ and GPTQ quantized large language models, covering environment setup, calibration data preparation, model quantization, deployment scripts, performance benchmarking, accuracy checks, best‑practice recommendations, and troubleshooting tips for GPU memory optimization.

AWQGPTQGPU memory optimization
0 likes · 45 min read
vLLM Quantized Inference: Loading AWQ/GPTQ Models and Optimizing GPU Memory
21CTO
21CTO
Jun 27, 2026 · Artificial Intelligence

Large vs Small Language Models: An Apple‑Centric Technical Comparison

The article analyses how deployment targets, inference economics, and training budgets drive divergent design choices for large (LLM) and small (SLM) Transformer‑based language models, covering architecture tweaks, data‑centric training methods, quantisation, KV‑cache management, and hybrid routing strategies for production systems.

Hybrid InferenceLarge Language ModelsSmall language models
0 likes · 16 min read
Large vs Small Language Models: An Apple‑Centric Technical Comparison
AI Programming Lab
AI Programming Lab
Jun 26, 2026 · Industry Insights

How to Land a Role in a Cutting‑Edge AI Lab Amid DeepSeek’s Hiring Surge

The article breaks down DeepSeek’s massive hiring push, outlines the core traits DeepMind’s Vlad Feinberg says are essential, and provides a step‑by‑step roadmap—including kernel work, scaling‑law studies, and open‑source contributions—to help aspiring engineers break into top AI research labs.

AI labsDeepSeekjob preparation
0 likes · 10 min read
How to Land a Role in a Cutting‑Edge AI Lab Amid DeepSeek’s Hiring Surge
DataFunSummit
DataFunSummit
Jun 25, 2026 · Cloud Computing

From Text to Images: Building Multi‑Modal Product Search with Elasticsearch Serverless

The article walks through the evolution of e‑commerce search from simple keyword matching to multi‑modal retrieval, explains a generic architecture that fuses text and image embeddings, details core techniques such as dense, sparse and hybrid models, vector similarity metrics, quantization methods like SQ and BBQ, and demonstrates how Elasticsearch Serverless provides a server‑less, cost‑effective platform to implement the end‑to‑end solution.

AIElasticsearchEmbedding
0 likes · 21 min read
From Text to Images: Building Multi‑Modal Product Search with Elasticsearch Serverless
Raymond Ops
Raymond Ops
Jun 22, 2026 · Artificial Intelligence

Elastic Deployment and GPU Scheduling for Large‑Model Inference with vLLM on Kubernetes

This article presents a detailed, step‑by‑step analysis of deploying the high‑performance vLLM inference engine on Kubernetes, covering GPU memory management, tensor parallelism, quantization choices, continuous batching, and automated scaling with HPA/KEDA to achieve low latency and high throughput for large language models.

DockerGPU schedulingKubernetes
0 likes · 49 min read
Elastic Deployment and GPU Scheduling for Large‑Model Inference with vLLM on Kubernetes
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 19, 2026 · Artificial Intelligence

Gemma‑4‑12B‑v2 (Fable 5 Clone) Achieves 3.5× Telecom Benchmark Boost

The author reproduces Anthropic’s Fable 5 using Gemma‑4‑12B‑v2, showing a 3.5× improvement on the telecom tau2‑bench versus the base model, details the agentic, coding, and general training data, compares quantization sizes, provides llama.cpp launch commands, and notes speed gains from speculative MTP decoding and current limitations.

Fable 5Gemma-4-12Bagentic AI
0 likes · 9 min read
Gemma‑4‑12B‑v2 (Fable 5 Clone) Achieves 3.5× Telecom Benchmark Boost
DataFunSummit
DataFunSummit
Jun 18, 2026 · Artificial Intelligence

From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless

This article examines the evolution of e‑commerce search from simple keyword matching to multimodal, cross‑modal retrieval, explains the core embedding and vector‑search technologies, compares dense, sparse and hybrid models, and demonstrates how Elasticsearch Serverless and Alibaba Cloud AI Search Platform enable a low‑cost, serverless, high‑performance end‑to‑end multimodal product search solution.

AI search platformBBQElasticsearch Serverless
0 likes · 21 min read
From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless
ThinkingAgent
ThinkingAgent
Jun 17, 2026 · Artificial Intelligence

Why a 7B Model Can Outperform a 70B Model: The Power of Knowledge Distillation

The article explains how knowledge distillation lets a small LLM learn from a much larger teacher model, achieving near‑teacher performance while cutting inference cost, latency and memory, and provides a step‑by‑step guide, benchmark results, advanced on‑policy techniques, common pitfalls and best‑practice recommendations.

LLMLLaMA-3On-Policy Distillation
0 likes · 13 min read
Why a 7B Model Can Outperform a 70B Model: The Power of Knowledge Distillation
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 1, 2026 · Artificial Intelligence

Opus‑Distilled Qwen3.5‑Coder Scores 100/100 Tool Calls, 1.4‑2.2× Faster with MTP, 128K Context on Consumer GPU

The article introduces Qwopus3.5‑4B‑Coder‑MTP‑GGUF, a 4‑billion‑parameter agent model fine‑tuned for code debugging, tool calling, and structured reasoning, explains its novel Trace Inversion, high‑quality trajectory data, and Curriculum SFT training, details MTP acceleration, benchmark results, quantization options, and step‑by‑step local deployment instructions.

AgentGGUFMTP
0 likes · 10 min read
Opus‑Distilled Qwen3.5‑Coder Scores 100/100 Tool Calls, 1.4‑2.2× Faster with MTP, 128K Context on Consumer GPU
Old Zhang's AI Learning
Old Zhang's AI Learning
May 31, 2026 · Artificial Intelligence

Qwen3.6-35B-A3B NVFP4: A Stable, Highly Compressed Quantized Model

NVIDIA's NVFP4 quantization reduces Qwen3.6-35B-A3B's memory footprint by threefold with almost no accuracy loss, offers plug‑and‑play deployment via vLLM, and outperforms other 4‑bit formats on Hopper/Blackwell GPUs, making it a practical choice for production AI workloads.

MoENVFP4Qwen3.6-35B-A3B
0 likes · 13 min read
Qwen3.6-35B-A3B NVFP4: A Stable, Highly Compressed Quantized Model
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
May 29, 2026 · Artificial Intelligence

How Alibaba Cloud Milvus Achieves 20× Faster Billion‑Scale Vector Search with DiskANN and RaBitQ

Alibaba Cloud Milvus combines DiskANN graph indexing with the RaBitQ quantization algorithm, delivering over 20× higher QPS, sub‑10% P99 latency, 29% lower memory usage and more than 98% recall on a 100 million‑vector, 768‑dimensional benchmark, while also cutting index build time from 20 h to about 6 h.

DiskANNMilvusRaBitQ
0 likes · 7 min read
How Alibaba Cloud Milvus Achieves 20× Faster Billion‑Scale Vector Search with DiskANN and RaBitQ
DataFunSummit
DataFunSummit
May 27, 2026 · Artificial Intelligence

From Text to Images: Building Multi‑Modal Product Search with Elasticsearch Serverless

This article walks through a complete multi‑modal product search solution that transforms textual and visual product data into embeddings, leverages dense, sparse and hybrid models, applies vector similarity and quantization techniques such as SQ and BBQ, and demonstrates how Elasticsearch Serverless provides a serverless, cost‑effective, auto‑scaling backbone for end‑to‑end retrieval.

AI Search Open PlatformElasticsearch ServerlessEmbedding
0 likes · 22 min read
From Text to Images: Building Multi‑Modal Product Search with Elasticsearch Serverless
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 22, 2026 · Artificial Intelligence

20‑Year‑Old Transformer Co‑author Open‑Sources a 218‑Billion‑Parameter Model

Cohere’s Command A+ model, built by Transformer co‑author Aidan Gomez and backed by Nick Frosst, packs 218 billion parameters but activates only 25 billion at inference, uses a lossless 4‑bit quantization scheme, offers native citation support, runs on a single B200 or two H100 GPUs, and is released under an Apache 2.0 license, marking a major shift toward truly open‑source, enterprise‑ready large language models.

AIApache 2.0Cohere
0 likes · 12 min read
20‑Year‑Old Transformer Co‑author Open‑Sources a 218‑Billion‑Parameter Model
Old Zhang's AI Learning
Old Zhang's AI Learning
May 17, 2026 · Artificial Intelligence

Why DeepSeek V4 Flash’s Quantized Model Is Gaining Traction

The DeepSeek V4 Flash quantized GGUF model and the dedicated ds4 inference engine, both released by antirez, offer dramatically reduced activation parameters, massive 1‑million‑token context windows, aggressive KV‑cache compression and hardware‑specific quantizations that enable smooth local inference on high‑memory Macs and CUDA machines, while sacrificing generality for performance.

DS4DeepSeek-V4-FlashGGUF
0 likes · 11 min read
Why DeepSeek V4 Flash’s Quantized Model Is Gaining Traction
DataFunSummit
DataFunSummit
May 15, 2026 · Artificial Intelligence

From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless

The article analyzes the shift from keyword‑based to multimodal e‑commerce search, outlines a generic architecture that combines text and image embedding with vector retrieval, and demonstrates how Elasticsearch Serverless and Alibaba Cloud AI Search platform enable a low‑cost, scalable, and high‑performance product search solution.

AI SearchElasticsearchEmbedding
0 likes · 20 min read
From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless
SuanNi
SuanNi
May 13, 2026 · Artificial Intelligence

How MiniCPM-V 4.6 Achieves Lightning‑Fast Multimodal AI on Smartphones (Open‑Source)

MiniCPM-V 4.6 combines a SigLIP2 visual encoder with a Qwen3.5 LLM, cuts FLOPs by over 50%, lowers token cost up to 43×, scores 13 on the Artificial Analysis Intelligence Index, and runs with 75 ms first‑token latency on 3136×3136 images across iOS, Android and HarmonyOS, all with fully open‑source code and extensive quantization support.

MiniCPM-VOpen Sourcebenchmark
0 likes · 6 min read
How MiniCPM-V 4.6 Achieves Lightning‑Fast Multimodal AI on Smartphones (Open‑Source)
Lao Guo's Learning Space
Lao Guo's Learning Space
May 12, 2026 · Artificial Intelligence

Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek

This article breaks down the key algorithms that power large‑language models—Transformer, Mixture‑of‑Experts, Flash Attention, KV‑Cache, Multi‑Token Prediction, quantization, Chain‑of‑Thought and Retrieval‑Augmented Generation—explaining how each contributes to the performance of ChatGPT, GPT‑4 and DeepSeek.

Flash AttentionKV cacheLarge Language Models
0 likes · 10 min read
Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek
DataFunSummit
DataFunSummit
May 7, 2026 · Artificial Intelligence

From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless

This article walks through a complete multimodal product search solution, explaining how embedding and vector retrieval technologies—combined with Elasticsearch Serverless and Alibaba Cloud AI Search—enable image‑based and semantic queries, detailing the architecture, key algorithms, quantization tricks, and practical deployment steps.

AI SearchElasticsearchEmbedding
0 likes · 22 min read
From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless
Machine Heart
Machine Heart
May 6, 2026 · Artificial Intelligence

Turning Your Mac into a Private AI Workstation with Cider and Mano‑P

The article analyzes how Ollama's shift to Apple’s MLX framework unlocks major speed gains on M5‑class Macs, then introduces the open‑source Cider inference accelerator and Mano‑P visual agent, detailing their quantization modes, benchmark results, hardware constraints, and how together they enable fast, offline private AI on macOS.

Apple SiliconCiderMLX
0 likes · 15 min read
Turning Your Mac into a Private AI Workstation with Cider and Mano‑P
Architects' Tech Alliance
Architects' Tech Alliance
May 1, 2026 · Artificial Intelligence

How DeepSeek V4 Triggers a Global AI Price War with OpenAI

DeepSeek V4’s open‑source 1 M‑token MoE model delivers benchmark scores of MMLU 88.7, C‑Eval 92.1 and HumanEval 69.5, while its 4‑bit AWQ quantization, PagedAttention memory management and FlashAttention acceleration cut inference costs and latency, prompting rivals such as Anthropic, OpenAI, Baidu and Huawei to slash prices and boost efficiency in a fierce market battle.

AI efficiencyDeepSeek V4MoE
0 likes · 9 min read
How DeepSeek V4 Triggers a Global AI Price War with OpenAI
AI Engineer Programming
AI Engineer Programming
Apr 25, 2026 · Artificial Intelligence

Quantization Across Signal Processing, AI Inference, and RAG Vector Search

This article explains how quantization—originating from signal processing—reduces precision to save resources, details its application to neural network weights and activations via PTQ, QAT, GPTQ, AWQ, and SmoothQuant, and shows how vector quantization enables fast, memory‑efficient retrieval in large‑scale RAG systems.

AWQGPTQLLM
0 likes · 19 min read
Quantization Across Signal Processing, AI Inference, and RAG Vector Search
Woodpecker Software Testing
Woodpecker Software Testing
Apr 24, 2026 · Artificial Intelligence

Practical Guide to Optimizing Large Model Performance in Production

This guide details how enterprises can move large language models from lab to production by defining specific SLI/SLO metrics, diagnosing hidden bottlenecks such as tokenizer latency, and applying four quantifiable optimization levers that dramatically improve latency, throughput, and cost efficiency.

GPU OptimizationLarge Language ModelsLoRA
0 likes · 6 min read
Practical Guide to Optimizing Large Model Performance in Production
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 19, 2026 · Artificial Intelligence

Qwen3.6-35B: 4‑bit Quantization, DFlash Speedup, Claude Opus Distillation

The article reviews three optimization paths for the Qwen3.6‑35B model—four‑bit AWQ quantization variants, the DFlash speculative decoding accelerator, and a Claude Opus‑based distillation—detailing their implementation steps, benchmark results, and guidance on selecting the best version for different hardware and performance needs.

AIDFlashQwen3.6
0 likes · 11 min read
Qwen3.6-35B: 4‑bit Quantization, DFlash Speedup, Claude Opus Distillation
DataFunSummit
DataFunSummit
Apr 19, 2026 · Artificial Intelligence

How to Build a Multimodal Product Search Engine with Embedding and Vector Retrieval on Elasticsearch Serverless

This article explains a complete multimodal product search solution that combines text and image embeddings, dense, sparse, and hybrid models, vector similarity metrics, and Elasticsearch Serverless features such as dense_vector, sparse_vector, hybrid search, quantization, and RRF ranking to achieve fast, accurate, and cost‑effective retrieval.

AIElasticsearchEmbedding
0 likes · 20 min read
How to Build a Multimodal Product Search Engine with Embedding and Vector Retrieval on Elasticsearch Serverless
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 18, 2026 · Artificial Intelligence

How to Run MiniMax‑M2.7 on Mac: Comparing Two Quantization Paths

This article explains why standard uniform quantization fails for the 228‑billion‑parameter MiniMax‑M2.7 MoE model on macOS, and compares two practical solutions—JANGTQ + MLX Studio with 2‑bit mixed‑precision achieving 91.5 % MMLU using 56.5 GB, and LM Studio + GGUF which is easier but requires at least 138 GB RAM and yields lower accuracy.

JANGTQLM‑StudioMLX Studio
0 likes · 8 min read
How to Run MiniMax‑M2.7 on Mac: Comparing Two Quantization Paths
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 12, 2026 · Artificial Intelligence

How to Deploy MiniMax-M2.7 Quantized Models Locally on macOS and Linux

This guide explains the 22 GGUF quantized versions of MiniMax-M2.7 released by Unsloth, compares their accuracy and size, recommends the UD‑Q4_K_XL model for best quality‑to‑size trade‑off, and provides step‑by‑step instructions for local deployment via Unsloth Studio, llama.cpp, API server, or the MLX native solution, along with important pitfalls and performance‑tuning tips.

Dynamic 2.0MLXMiniMax M2.7
0 likes · 14 min read
How to Deploy MiniMax-M2.7 Quantized Models Locally on macOS and Linux
AI Tech Publishing
AI Tech Publishing
Apr 9, 2026 · Artificial Intelligence

Engineering‑Focused Guide to Training and Inference of Large Language Models

This article walks engineers through the full LLM stack—from tokenization and positional encoding to transformer blocks, efficient fine‑tuning, quantization, and production‑grade inference techniques such as KV‑cache, FlashAttention, PagedAttention, continuous batching, and speculative decoding—highlighting trade‑offs, toolchains, and practical workflow steps.

Fine-tuningLLMLoRA
0 likes · 13 min read
Engineering‑Focused Guide to Training and Inference of Large Language Models
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Apr 8, 2026 · Artificial Intelligence

Dissecting Gemma‑4’s Architecture and Training Choices: A Technical Comparison with Qwen‑3 and GLM‑5

This article breaks down every architectural and training decision behind Gemma‑4—KV sharing, p‑RoPE, per‑layer embeddings, and a dual‑path MoE + dense MLP—while contrasting its efficiency and performance with Qwen‑3 and GLM‑5 across benchmarks, quantization strategies, and RL pipelines.

GLM-5Gemma 4LLM architecture
0 likes · 23 min read
Dissecting Gemma‑4’s Architecture and Training Choices: A Technical Comparison with Qwen‑3 and GLM‑5
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Apr 8, 2026 · Artificial Intelligence

Unlocking 8‑Hour Autonomous Coding: GLM‑5.1’s Leap with Kunlun XPU

The open‑source GLM‑5.1 model, adapted to Baidu Baige's Kunlun XPU via the vLLM‑Kunlun Plugin, delivers record‑breaking SWE‑bench scores, eight‑hour autonomous coding, long‑context handling up to 64K tokens, and scalable deployment across tens of thousands of chips, showcasing end‑to‑end AI acceleration.

GLM-5.1Kunlun XPUModel deployment
0 likes · 8 min read
Unlocking 8‑Hour Autonomous Coding: GLM‑5.1’s Leap with Kunlun XPU
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 1, 2026 · Artificial Intelligence

Running Large Models Locally on Mac: The Most Powerful Current Solution

This article reviews the JANG quantization format, the vMLX inference engine with a five‑layer cache stack, and the MLX Studio GUI, showing how their combination enables 397B‑parameter models to fit on 128 GB Apple Silicon Macs, achieve up to 224× faster first‑token latency for 100K context, and provide a full‑featured local AI experience.

Apple SiliconJANGLarge Language Models
0 likes · 8 min read
Running Large Models Locally on Mac: The Most Powerful Current Solution
SuanNi
SuanNi
Mar 29, 2026 · Industry Insights

Did Google’s TurboQuant Steal RaBitQ? Unpacking the AI Compression Controversy

The article examines Google’s TurboQuant compression breakthrough, its claimed 6‑fold KV cache reduction and 8× speedup, and the allegations that it mirrors the earlier RaBitQ method, detailing technical similarities, disputed experiments, market fallout, and the ongoing academic debate.

AIacademic integrityindustry impact
0 likes · 11 min read
Did Google’s TurboQuant Steal RaBitQ? Unpacking the AI Compression Controversy
DataFunSummit
DataFunSummit
Mar 29, 2026 · Artificial Intelligence

How to Build a Multimodal Product Search Engine with Embedding and Vector Retrieval on Elasticsearch Serverless

This article explores the evolution of e‑commerce search toward multimodal and cross‑modal capabilities, outlines a generic architecture that combines text and image processing via embedding and vector retrieval, and demonstrates how to implement the solution using Alibaba Cloud's AI Search Open Platform and Elasticsearch Serverless with detailed guidance on models, similarity metrics, quantization, and performance optimization.

AIElasticsearchEmbedding
0 likes · 22 min read
How to Build a Multimodal Product Search Engine with Embedding and Vector Retrieval on Elasticsearch Serverless
AI Engineer Programming
AI Engineer Programming
Mar 28, 2026 · Artificial Intelligence

How to Start Training Your Own AI Model: A Complete Roadmap

This guide maps the end-to-end process for building a small AI model—from leveraging open-source base models and applying SFT with LoRA/QLoRA, through alignment techniques like DPO or ORPO, to low-cost distillation and final quantization for local deployment, while recommending free GPU resources and essential tooling.

AIAlignmentLoRA
0 likes · 12 min read
How to Start Training Your Own AI Model: A Complete Roadmap
DataFunSummit
DataFunSummit
Mar 24, 2026 · Artificial Intelligence

How to Build a Multimodal Product Search System with Embedding and Vector Retrieval

This article presents a comprehensive, end‑to‑end solution for multimodal product search, detailing the evolution from keyword to image‑based queries, the core embedding and vector retrieval technologies, practical Elasticsearch Serverless integration, quantization methods, and a complete demo workflow for building a high‑performance, low‑cost search platform.

AI search platformElasticsearchEmbedding
0 likes · 21 min read
How to Build a Multimodal Product Search System with Embedding and Vector Retrieval
Woodpecker Software Testing
Woodpecker Software Testing
Mar 17, 2026 · Artificial Intelligence

5 Proven Strategies to Boost Large Language Model Performance

The article presents five actionable strategies—defining a three‑dimensional performance baseline, applying layered injection load tests, co‑optimizing dynamic quantization with cache, employing SLO‑driven chaos engineering, and shifting testing left to compilation—to reliably measure and improve LLM throughput, latency, and resource efficiency in production.

Chaos EngineeringLLM OptimizationLarge Language Models
0 likes · 7 min read
5 Proven Strategies to Boost Large Language Model Performance
Old Zhang's AI Learning
Old Zhang's AI Learning
Feb 26, 2026 · Artificial Intelligence

Ultimate Guide to Local Deployment of Qwen3.5 Models (27B‑397B)

This guide reviews the Qwen3.5 model lineup, explains mixed‑inference and MoE architecture, presents benchmark comparisons with GPT‑5.2, Claude 4.5 and Gemini‑3 Pro, evaluates 4‑bit and 3‑bit quantization loss, outlines hardware requirements, and provides step‑by‑step deployment options using llama.cpp or llama‑server.

MoEQwen3.5inference
0 likes · 14 min read
Ultimate Guide to Local Deployment of Qwen3.5 Models (27B‑397B)
Past Memory Big Data
Past Memory Big Data
Feb 25, 2026 · Artificial Intelligence

How Google’s TPU Systolic Array Powered AlphaGo and Large Language Models

Google’s Tensor Processing Unit (TPU) uses a systolic array architecture and low‑precision quantization to overcome the Von Neumann bottleneck, delivering orders‑of‑magnitude higher throughput and energy efficiency for matrix‑multiplication‑heavy AI workloads—from AlphaGo’s inference to today’s massive language models.

AI hardwareGoogleSystolic Array
0 likes · 15 min read
How Google’s TPU Systolic Array Powered AlphaGo and Large Language Models
Weekly Large Model Application
Weekly Large Model Application
Feb 22, 2026 · Artificial Intelligence

2026 Guide: Pure‑CPU Open‑Source Chinese TTS Models Optimized for Performance

This article reviews the most capable open‑source Chinese text‑to‑speech models that run entirely on CPU in 2026, compares their quantization and speed features, recommends acceleration engines, outlines five hard‑won optimization rules, and provides a concise selection guide for various deployment scenarios.

CPU inferenceChinese TTSONNX Runtime
0 likes · 6 min read
2026 Guide: Pure‑CPU Open‑Source Chinese TTS Models Optimized for Performance
Weekly Large Model Application
Weekly Large Model Application
Feb 22, 2026 · Artificial Intelligence

2026 Guide to Running Open‑Source ASR on Pure CPU

The 2026 overview details lightweight, heavily quantized open‑source speech‑recognition models and CPU‑specific inference engines, offering concrete tips, model comparisons, and a concise selection guide that enable real‑time, GPU‑free ASR deployment with low latency and high stability.

ASRCPU inferenceModel Selection
0 likes · 4 min read
2026 Guide to Running Open‑Source ASR on Pure CPU
Old Zhang's AI Learning
Old Zhang's AI Learning
Feb 16, 2026 · Artificial Intelligence

A New Extreme Quantization Tool for Large Models: AngelSlim’s 2‑Bit Compression

AngelSlim introduces a full‑stack large‑model compression suite that uses quantization‑aware training to shrink a 1.8B LLM to 2‑bit precision, achieving less than 4% accuracy loss, supporting a wide range of models, speculative decoding, and providing end‑to‑end deployment instructions for MacBook M4 and server environments.

AngelSlimGGUFLarge Language Models
0 likes · 13 min read
A New Extreme Quantization Tool for Large Models: AngelSlim’s 2‑Bit Compression
AI Engineering
AI Engineering
Feb 15, 2026 · Artificial Intelligence

Qwen3‑ASR Runs Natively on Apple Silicon via MLX for Full‑Speed Speech Recognition

A developer has re‑implemented the state‑of‑the‑art Qwen3‑ASR model in MLX, enabling native execution on Apple M1‑M4 chips with real‑time factors as low as 0.08, 4‑bit quantization speedups of 4.7×, multilingual support for 52 languages, and features such as word‑level timestamps and streaming transcription.

Apple SiliconMLXQwen3-ASR
0 likes · 5 min read
Qwen3‑ASR Runs Natively on Apple Silicon via MLX for Full‑Speed Speech Recognition
Old Zhang's AI Learning
Old Zhang's AI Learning
Feb 12, 2026 · Artificial Intelligence

Testing the World's Most Powerful Open‑Source LLM: GLM‑5, Local Deployment & Free Ollama Cloud

The article evaluates GLM‑5, the claimed strongest open‑source large language model, comparing its benchmark scores to Claude Opus, Gemini and GPT, detailing its DeepSeek‑inspired architecture, quantized FP8 deployment requirements, and step‑by‑step usage of Ollama’s free cloud model with Agent, data‑analysis and document‑generation features.

AI benchmarkingAgent modeData Analysis
0 likes · 7 min read
Testing the World's Most Powerful Open‑Source LLM: GLM‑5, Local Deployment & Free Ollama Cloud
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Feb 4, 2026 · Artificial Intelligence

Google’s Second Sword: Accelerating LLM Inference with Speculative Decoding and Cascades

The article analyzes Google’s shift from scaling‑law to efficiency‑law, detailing how speculative decoding, language‑model cascades, distillation, CALM, accurate quantized training, and the Mixture‑of‑Recursions architecture together form a multi‑layered strategy to cut inference cost, boost throughput, and sustain the company’s AI moat.

Google TPUInference AccelerationLanguage Model Cascades
0 likes · 8 min read
Google’s Second Sword: Accelerating LLM Inference with Speculative Decoding and Cascades
AI Waka
AI Waka
Feb 1, 2026 · Artificial Intelligence

Boost LLM Inference Speed: Precision Tricks, Quantization, and Multi‑GPU Strategies

This article reviews practical techniques for accelerating large language model inference—including reduced‑precision formats, post‑training quantization, adapter‑based fine‑tuning, pruning, continuous batch processing, and multi‑GPU deployment—while providing concrete code examples, benchmark results, and guidance on selecting the right approach for production workloads.

GPULLMadapter-fine-tuning
0 likes · 20 min read
Boost LLM Inference Speed: Precision Tricks, Quantization, and Multi‑GPU Strategies
AI Cyberspace
AI Cyberspace
Jan 26, 2026 · Artificial Intelligence

How NVFP4 Quantization Supercharges LLM Inference on NVIDIA DGX

This article explains the NVFP4 4‑bit floating‑point quantization technique, shows how to deploy Qwen3‑30B‑A3B models with TensorRT‑LLM and vLLM, compares performance across NVFP4, AWQ and INT8 quantizations, and provides practical profiling commands for NVIDIA DGX systems.

LLMNVFP4NVIDIA DGX
0 likes · 23 min read
How NVFP4 Quantization Supercharges LLM Inference on NVIDIA DGX
PaperAgent
PaperAgent
Jan 17, 2026 · Artificial Intelligence

How Qwen3‑VL Embedding and Reranker Set New SOTA in Multimodal Retrieval

The article analyzes the Qwen3‑VL‑Embedding and Qwen3‑VL‑Reranker models, detailing their unified vector space, multi‑stage training pipeline, Matryoshka representation learning, quantization techniques, massive synthetic data generation, and benchmark results that push multimodal retrieval performance to a new state‑of‑the‑art.

Embeddingknowledge distillationlarge language model
0 likes · 7 min read
How Qwen3‑VL Embedding and Reranker Set New SOTA in Multimodal Retrieval
MaGe Linux Operations
MaGe Linux Operations
Dec 27, 2025 · Artificial Intelligence

How to Deploy and Optimize Enterprise‑Scale LLM Inference Services: A Practical Guide

This guide walks you through deploying large language models such as ChatGLM and Llama in production, covering environment setup, model quantization, dynamic batching, service configuration, Nginx load balancing, monitoring, troubleshooting, and best‑practice recommendations for high‑performance, cost‑effective AI inference.

GPULLMinference
0 likes · 48 min read
How to Deploy and Optimize Enterprise‑Scale LLM Inference Services: A Practical Guide
DataFunSummit
DataFunSummit
Dec 20, 2025 · Artificial Intelligence

How AutoHome Built the Cangjie Large Model: From Training Architecture to Real-World AI Applications

This article details AutoHome's end‑to‑end development of the Cangjie large model, covering the training infrastructure with distributed data, pipeline and tensor parallelism, core business use cases such as video script generation and multi‑tool Agent capabilities, inference optimizations through quantization and fast serving frameworks, and future directions for personalized automotive AI services.

Agent AIDistributed TrainingVideo Generation
0 likes · 19 min read
How AutoHome Built the Cangjie Large Model: From Training Architecture to Real-World AI Applications
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 18, 2025 · Artificial Intelligence

How to Build a Real‑Time AI‑Powered Anime‑Style Video Generator for Social Apps

This technical report details the end‑to‑end workflow for integrating an AIGC video generation module into a social app, covering requirement analysis, model and hardware selection, dataset construction, LoRA and full‑parameter training, multiple acceleration techniques such as Sage Attention, TeaCache, XDiT, gradient‑checkpointing offload, tiled VAE, and quantization, followed by extensive performance evaluation and metric‑based ranking of the final models.

AI video generationLoRA fine-tuningModel Optimization
0 likes · 38 min read
How to Build a Real‑Time AI‑Powered Anime‑Style Video Generator for Social Apps
Data Party THU
Data Party THU
Nov 2, 2025 · Operations

How to Maximize vLLM Throughput: Batch Size, Quantization, and Monitoring Tips

This guide explains how to unleash vLLM’s full potential by optimizing batch size, leveraging 4‑bit quantization, tuning concurrency parameters, planning capacity with token‑per‑second metrics, and implementing robust monitoring to balance latency, cost, and scalability in production deployments.

LLM servingMonitoringbatching
0 likes · 10 min read
How to Maximize vLLM Throughput: Batch Size, Quantization, and Monitoring Tips
Meituan Technology Team
Meituan Technology Team
Oct 15, 2025 · Artificial Intelligence

What’s New in Large Model Research? Top Meituan AI Papers Up to Oct 2025

This curated list showcases Meituan’s latest large‑model breakthroughs and academic papers up to October 2025, spanning LLM system optimizations, multimodal generation, evaluation benchmarks, quantization techniques, and reinforcement‑learning‑driven improvements, offering researchers valuable insights and resources across the AI landscape.

AI researchBenchmarkingLarge Language Models
0 likes · 10 min read
What’s New in Large Model Research? Top Meituan AI Papers Up to Oct 2025
Hailey Says
Hailey Says
Sep 7, 2025 · Artificial Intelligence

What to Consider When Discussing Large AI Models

The article explains how to choose suitable large AI models for product use by focusing on three key aspects—model inference, post‑training optimization techniques such as quantization, pruning and distillation, and the role of GPUs, including their kernels, software stack, and computing platforms.

GPUPruninginference
0 likes · 8 min read
What to Consider When Discussing Large AI Models
Data Party THU
Data Party THU
Sep 4, 2025 · Artificial Intelligence

How MXFP4 Quantization Lets a 1200‑Billion‑Parameter LLM Run on a Single 80GB GPU

This article analyzes the memory bottleneck of massive language models, explains the mathematical modeling of memory requirements, evaluates traditional sharding limits, and details how GPT‑OSS’s MXFP4 quantization combined with Mixture‑of‑Experts reduces memory, bandwidth, and compute demands enough to fit a 1200‑billion‑parameter model onto an 80 GB GPU with minimal accuracy loss.

FP4LLMMXFP4
0 likes · 11 min read
How MXFP4 Quantization Lets a 1200‑Billion‑Parameter LLM Run on a Single 80GB GPU
AI Algorithm Path
AI Algorithm Path
Aug 23, 2025 · Artificial Intelligence

Understanding QAT: Quantization‑Aware Training with PyTorch

This article explains the principles of model quantization, compares post‑training quantization (PTQ) and quantization‑aware training (QAT), details the QAT workflow in PyTorch—including fake quantization, gradient handling, and code examples—and offers practical tips for achieving high‑accuracy int8/int4 models.

Fake QuantizationPyTorchQAT
0 likes · 15 min read
Understanding QAT: Quantization‑Aware Training with PyTorch
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 11, 2025 · Artificial Intelligence

How Multimodal Product Search Transforms E‑Commerce with Embedding and Vector Retrieval

This article explores the evolution from keyword‑based to multimodal e‑commerce search, detailing a universal solution that combines text and image processing through embedding and vector retrieval, and demonstrates how Alibaba Cloud's AI Search Open Platform and Elasticsearch Serverless enable fast, low‑cost, and scalable multimodal product search deployments.

Embeddingmultimodal searchquantization
0 likes · 17 min read
How Multimodal Product Search Transforms E‑Commerce with Embedding and Vector Retrieval
21CTO
21CTO
Jul 22, 2025 · Artificial Intelligence

Run Powerful LLMs Locally on <8GB RAM: Top 10 Small Models & Tools

This article explains how advanced quantization and model optimization enable running strong large language models on laptops or desktops with less than 8 GB of RAM or VRAM, outlines key technical concepts, recommends local inference tools, and lists ten compact LLMs with usage commands.

AILLM toolsLocal LLM
0 likes · 10 min read
Run Powerful LLMs Locally on <8GB RAM: Top 10 Small Models & Tools
AI Algorithm Path
AI Algorithm Path
Jul 19, 2025 · Artificial Intelligence

Understanding LoRA and QLoRA: Techniques for Efficient LLM Fine‑Tuning

This article explains how low‑rank adaptation (LoRA) and its quantized variant (QLoRA) compress large language model weights, reduce training cost, and enable flexible adapter switching, while detailing matrix decomposition, training mechanics, and trade‑offs with concrete examples and quantitative analysis.

LLM fine-tuningLoRAQLoRA
0 likes · 11 min read
Understanding LoRA and QLoRA: Techniques for Efficient LLM Fine‑Tuning
AI Algorithm Path
AI Algorithm Path
Jul 13, 2025 · Artificial Intelligence

How to Calculate the Right AI Model Size for Your PC (3B, 7B, 13B)

This article explains how to estimate the GPU memory required for running large language models of 3 B, 7 B, and 13 B parameters, walks through step‑by‑step calculations, shows how hardware limits affect feasibility, and offers practical optimization techniques such as quantization and CPU offloading.

AI model sizingCPU offloadingFP16
0 likes · 5 min read
How to Calculate the Right AI Model Size for Your PC (3B, 7B, 13B)
Tencent Technical Engineering
Tencent Technical Engineering
Jul 3, 2025 · Artificial Intelligence

Winning the NTIRE 2025 UGC Video Enhancement Challenge: A Progressive AI Framework

Tencent’s TEG team secured first place in the NTIRE 2025 UGC Video Enhancement competition by introducing a progressive, three‑stage AI framework that decomposes enhancement tasks into expert models for color correction, denoising, and temporal stability, incorporates advanced loss functions, extensive hardware‑level optimizations, INT8 quantization techniques, and outlines future diffusion‑based generative enhancements.

AIdiffusion modelshardware optimization
0 likes · 17 min read
Winning the NTIRE 2025 UGC Video Enhancement Challenge: A Progressive AI Framework
Tencent Architect
Tencent Architect
Jul 2, 2025 · Artificial Intelligence

How Tencent’s TEG Shannon Lab Dominated the NTIRE 2025 UGC Video Enhancement Challenge

Tencent TEG Shannon Lab won the NTIRE 2025 UGC Video Enhancement competition with a progressive training framework that combines adaptive color enhancement, high‑speed denoising, and temporal stability under bitrate constraints, achieving top subjective scores, significant inference speed‑ups, and successful INT8 quantization for real‑time deployment.

AI video codecNTIRE2025Real-time inference
0 likes · 18 min read
How Tencent’s TEG Shannon Lab Dominated the NTIRE 2025 UGC Video Enhancement Challenge
AI Algorithm Path
AI Algorithm Path
Apr 22, 2025 · Artificial Intelligence

Understanding LLM Quantization: GPTQ, QAT, AWQ, GGUF, and GGML Explained

The article walks through the fundamentals of large‑language‑model quantization, presenting a concrete int8 example, detailed explanations of GPTQ, GGUF/GGML, QAT, and AWQ methods, and provides step‑by‑step code snippets, formulas, calibration procedures, and performance observations for each technique.

AWQGGMLGGUF
0 likes · 15 min read
Understanding LLM Quantization: GPTQ, QAT, AWQ, GGUF, and GGML Explained
DeWu Technology
DeWu Technology
Apr 14, 2025 · Artificial Intelligence

Overview of Recent Large Language Model Quantization Techniques

The article surveys modern post‑training quantization approaches for large language models, detailing weight‑only and activation‑aware methods such as GPTQ, AWQ, HQQ, SmoothQuant, QuIP, QuaRot, SpinQuant, QQQ, QoQ, and FP8, and compares their precision levels, algorithmic steps, accuracy‑throughput trade‑offs, and implementation considerations for efficient inference.

AILLMmodel compression
0 likes · 32 min read
Overview of Recent Large Language Model Quantization Techniques
58 Tech
58 Tech
Apr 11, 2025 · Artificial Intelligence

Optimization of Multimodal Visual Large Model Inference: Pre‑processing, ViT TensorRT, CUDA Graphs, Tokenization, Prefix Cache, and Quantization

This report details a comprehensive set of optimizations for multimodal visual large‑model (VLM) inference—including image pre‑processing acceleration, TensorRT integration for the ViT module, CUDA‑Graph replay, token‑count reduction, prefix‑cache handling, and weight quantization—demonstrating up to three‑fold throughput gains while maintaining accuracy.

CUDA GraphMultimodalTensorRT
0 likes · 19 min read
Optimization of Multimodal Visual Large Model Inference: Pre‑processing, ViT TensorRT, CUDA Graphs, Tokenization, Prefix Cache, and Quantization
Meituan Technology Team
Meituan Technology Team
Apr 10, 2025 · Artificial Intelligence

Meituan's 10 Papers at CVPR 2025 and ICLR 2025

This article presents concise summaries of ten selected ICLR 2025 and CVPR 2025 papers covering LLM alignment, temporal‑decay DPO, joint‑embedding predictive architecture, 4‑bit quantization, token‑focused VQA, universal visual segmentation, document understanding, fine‑grained spatio‑temporal modeling, visual quality evaluation, and ultra‑high‑resolution diffusion, and also announces face‑to‑face and online sharing sessions hosted by Meituan.

CVPR 2025ICLR 2025Large Language Model Alignment
0 likes · 19 min read
Meituan's 10 Papers at CVPR 2025 and ICLR 2025
Baidu Tech Salon
Baidu Tech Salon
Mar 13, 2025 · Artificial Intelligence

How PaddlePaddle 3.0 Boosts Large‑Model Inference with 4‑Bit Quantization and MLA Optimizations

PaddlePaddle 3.0 introduces a full‑stack inference engine that supports FP8, INT8, and 4‑bit quantization for popular LLMs such as DeepSeek V3/R1, delivers up to 2× token throughput on a single H800 GPU, and provides detailed deployment scripts for single‑node and multi‑node setups, including MTP speculative decoding and SageAttention for long‑sequence acceleration.

DockerLarge Language ModelsMLA
0 likes · 13 min read
How PaddlePaddle 3.0 Boosts Large‑Model Inference with 4‑Bit Quantization and MLA Optimizations