The article shows how recent advances in models like Gemma‑4 and GPT‑OSS have turned local LLM inference on a 2022 M2 Mac into a practical, near‑click‑ready workflow, complete with Docker‑based agent setups, performance observations, and detailed configuration code.
The article explains how recent advances in LLM performance now allow developers to run sophisticated AI models locally on a 2022 M2 Mac using tools like LM Studio, Pi agent, and Docker, detailing model choices, setup steps, performance observations, and remaining limitations.
The article analyzes DeepSeek V4's migration to Huawei's Ascend 950PR chip and CANN framework, detailing three hardware‑level innovations, the CUDA‑to‑CANN transition, and the resulting 35× inference speed boost, 2.87× performance over Nvidia H20, and dramatic cost reductions for trillion‑parameter models.
Baidu’s Baige team successfully adapted the GLM‑4.x series language models to the Kunlun XPU platform by leveraging SGLang and the vLLM‑Kunlun plugin, employing agile adaptation, precision alignment with torch_xray, and extensive performance tuning to achieve GPU‑level accuracy and superior inference speed.
The article details the Attention‑FFN Disaggregation (AFD) technique used by Baidu Baige to separate self‑attention and feed‑forward network stages in DeepSeek‑V3 models, describing multi‑stage scheduling, three‑batch overlap, communication optimizations, and performance results that achieve up to 19% throughput improvement under a 100 ms SLO.
Hooop leverages HarmonyOS's on‑device AI and custom vision algorithms to provide real‑time, offline basketball training by detecting shots, analyzing trajectories, automatically clipping scoring clips, and tracking performance metrics without an internet connection.
LongCat-Flash, an open‑source Mixture‑of‑Experts model released by Meituan, leverages model‑system co‑design, PD‑disaggregation, SBO scheduling and large‑scale expert parallelism within the SGLang framework to deliver dramatically lower latency, higher throughput and cost‑effective inference for AI agents, with detailed deployment instructions provided.
This article details a complete engineering solution for bringing on‑device AI to WeChat mini‑programs, comparing TensorFlow.js and WeChat native inference, covering model conversion, package‑size optimization, integration steps, performance metrics, and a hybrid strategy that boosts recommendation click‑through rates by 30%.
This article explores how AI infrastructure has shifted from CPU‑centric designs to GPU‑driven architectures, detailing hardware evolution, software changes, and the engineering challenges of large‑model training and inference, while offering practical insights for traditional backend engineers transitioning to AI systems.
This article walks through the challenges of prompting small vision language models, demonstrates a conventional JSON‑based prompt, then shows how to define Pydantic models, embed their JSON schema into prompts, run inference with Qwen2.5‑VL, and visualize the structured results.
The article summarizes Zhihu’s technical talk on the ZhiLight large‑model inference framework, detailing model execution mechanisms, GPU load analysis, multi‑GPU parallel strategies, open‑source engine comparisons, compute‑communication overlap, quantization techniques, benchmark results, and future directions for scalable LLM deployment.
Compiled by Tsinghua University experts, this guide systematically analyzes the DeepSeek‑R1 inference model and DeepResearch platform, offering multi‑model comparisons, real‑world case studies, and end‑to‑end AI‑driven solutions from data collection to report generation for researchers.
The FlightVGM paper, awarded Best Paper at FPGA 2025, details a novel FPGA-based inference IP for video generation models that leverages time‑space activation sparsity, mixed‑precision DSP58 extensions, and adaptive scheduling to achieve up to 1.30× performance and 4.49× energy‑efficiency gains over a NVIDIA 3090 GPU while preserving model accuracy.
The article reviews DeepSeek's open‑source R1 and V3 large language models, highlighting their technical breakthroughs, cost advantages, expert opinions, industry adoption across chips, cloud services, and applications, and discusses future directions for model scaling, distillation, and AI competition.
HighService, Alibaba’s Pythonic AI service framework, accelerates large‑model inference and maximizes GPU utilization by separating CPU‑GPU processes, offering out‑of‑the‑box quantization, parallelism and caching, and dynamically reallocating idle GPUs across clusters through a master‑worker scheduler to keep online latency low while boosting offline throughput for diffusion and LLM workloads.
This article details how to train a YOLOv8‑pose model for license‑plate detection, convert it to TensorRT engine, implement C++ inference and preprocessing, expose the functionality via CGO to Go, and assemble a lightweight web service for real‑time plate recognition.
Device intelligence brings on-device reasoning and real-time inference to smartphones and IoT gateways, delivering low-latency, privacy-preserving, personalized services such as AR/VR enhancements and recommendation re-ranking, while confronting challenges of hardware fragmentation and model size, and complementing cloud AI through architectures like Hala’s MNN-based pipeline.
FastDeploy is an open‑source toolkit that standardizes AI model APIs and enables developers to deploy vision, NLP, and speech models on diverse hardware—including GPUs, CPUs, Jetson, ARM, and various NPUs—using just three lines of code or a single command, while delivering end‑to‑end performance optimizations.
By applying high‑performance operators, quantization, and AI compiler optimizations with Alibaba Cloud PAI Blade and Intel Xeon back‑ends, the address‑standardization service’s deep‑learning models achieved up to 4.11× faster end‑to‑end inference without sacrificing accuracy, enabling more complex models and lower latency.
The article details Meituan's engineering journey from small DNNs to hundred‑gigabyte deep learning models for food‑delivery ads, analyzing online latency and offline efficiency challenges and presenting distributed storage, CPU/GPU acceleration, OpenVINO, TensorRT, CodeGen, and data‑pipeline optimizations that dramatically improve throughput, memory usage, and sample‑building speed.
XGBoost Serving is an open‑source, high‑performance inference system built on TensorFlow Serving that adds dedicated servables for pure GBDT, GBDT+FM binary‑classification, and GBDT+FM multi‑classification models, providing automatic version lifecycle management, GRPC/HTTP APIs, and up to 50 % latency reduction, now available on GitHub after successful deployment in iQIYI’s recommendation platform.
