Tagged articles
model inference
24 articles · Page 1 of 1
A developer demonstrated that a 128 GB M5 Max Mac can stream‑load the 1.6 TB MXFP4 weights of the 2.8‑trillion‑parameter Kimi K3 model, achieving 0.32 token/s, while an 80‑GPU RTX 5090 cluster reaches 20 token/s, highlighting both feasibility and speed limits of large‑model inference on consumer hardware.
GPU clusterKimi K3Mac M5 Max
0 likes · 6 min read

The article walks through ten lesser‑known Codex settings—including browser capability, unlocking the highest inference strength, importing work from other AI apps, advanced computer control, quick‑snapshot shortcuts, the dormant GPT‑5.3‑Codex‑Spark mode, phone‑to‑Mac remote control, memory/Chronicle preview, and plugin tips—providing step‑by‑step paths and screenshots for each.
AI assistantCodexMemory
0 likes · 6 min read

The article shows how recent advances in models like Gemma‑4 and GPT‑OSS have turned local LLM inference on a 2022 M2 Mac into a practical, near‑click‑ready workflow, complete with Docker‑based agent setups, performance observations, and detailed configuration code.
DockerGemma 4LM Studio
0 likes · 12 min read

The article explains how recent advances in LLM performance now allow developers to run sophisticated AI models locally on a 2022 M2 Mac using tools like LM Studio, Pi agent, and Docker, detailing model choices, setup steps, performance observations, and remaining limitations.
AI developmentDockerGemma 4
0 likes · 10 min read

This article examines the hidden but costly strategic errors CTOs make when adopting AI—treating AI as a simple module, blindly building large models, ignoring AI‑native data architecture, applying traditional R&D management to AI teams, and postponing safety and compliance—offering a concrete self‑check checklist.
AI strategyCTOMoE routing
0 likes · 12 min read

The article analyzes DeepSeek V4's migration to Huawei's Ascend 950PR chip and CANN framework, detailing three hardware‑level innovations, the CUDA‑to‑CANN transition, and the resulting 35× inference speed boost, 2.87× performance over Nvidia H20, and dramatic cost reductions for trillion‑parameter models.
AI hardwareCANN frameworkDeepSeek
0 likes · 10 min read

Baidu’s Baige team successfully adapted the GLM‑4.x series language models to the Kunlun XPU platform by leveraging SGLang and the vLLM‑Kunlun plugin, employing agile adaptation, precision alignment with torch_xray, and extensive performance tuning to achieve GPU‑level accuracy and superior inference speed.
AIXPUhardware acceleration
0 likes · 6 min read

The article details the Attention‑FFN Disaggregation (AFD) technique used by Baidu Baige to separate self‑attention and feed‑forward network stages in DeepSeek‑V3 models, describing multi‑stage scheduling, three‑batch overlap, communication optimizations, and performance results that achieve up to 19% throughput improvement under a 100 ms SLO.
3BOAFDAttention-FFN Disaggregation
0 likes · 17 min read

Hooop leverages HarmonyOS's on‑device AI and custom vision algorithms to provide real‑time, offline basketball training by detecting shots, analyzing trajectories, automatically clipping scoring clips, and tracking performance metrics without an internet connection.
AIComputer VisionHarmonyOS
0 likes · 12 min read

LongCat-Flash, an open‑source Mixture‑of‑Experts model released by Meituan, leverages model‑system co‑design, PD‑disaggregation, SBO scheduling and large‑scale expert parallelism within the SGLang framework to deliver dramatically lower latency, higher throughput and cost‑effective inference for AI agents, with detailed deployment instructions provided.
LongCat-FlashSGLangcost efficient
0 likes · 15 min read

This article details a complete engineering solution for bringing on‑device AI to WeChat mini‑programs, comparing TensorFlow.js and WeChat native inference, covering model conversion, package‑size optimization, integration steps, performance metrics, and a hybrid strategy that boosts recommendation click‑through rates by 30%.
TensorFlow.jsWeChatmini-program
0 likes · 13 min read

This article explores how AI infrastructure has shifted from CPU‑centric designs to GPU‑driven architectures, detailing hardware evolution, software changes, and the engineering challenges of large‑model training and inference, while offering practical insights for traditional backend engineers transitioning to AI systems.
AI infrastructureGPU computingModel Training
0 likes · 16 min read

This article walks through the challenges of prompting small vision language models, demonstrates a conventional JSON‑based prompt, then shows how to define Pydantic models, embed their JSON schema into prompts, run inference with Qwen2.5‑VL, and visualize the structured results.
JSON SchemaPydanticPython
0 likes · 10 min read

The article summarizes Zhihu’s technical talk on the ZhiLight large‑model inference framework, detailing model execution mechanisms, GPU load analysis, multi‑GPU parallel strategies, open‑source engine comparisons, compute‑communication overlap, quantization techniques, benchmark results, and future directions for scalable LLM deployment.
GPU ParallelismSGLangZhiLight
0 likes · 11 min read

Compiled by Tsinghua University experts, this guide systematically analyzes the DeepSeek‑R1 inference model and DeepResearch platform, offering multi‑model comparisons, real‑world case studies, and end‑to‑end AI‑driven solutions from data collection to report generation for researchers.
AI researchData AutomationDeepSeek
0 likes · 6 min read

The FlightVGM paper, awarded Best Paper at FPGA 2025, details a novel FPGA-based inference IP for video generation models that leverages time‑space activation sparsity, mixed‑precision DSP58 extensions, and adaptive scheduling to achieve up to 1.30× performance and 4.49× energy‑efficiency gains over a NVIDIA 3090 GPU while preserving model accuracy.
AIFPGAhardware acceleration
0 likes · 11 min read

The article reviews DeepSeek's open‑source R1 and V3 large language models, highlighting their technical breakthroughs, cost advantages, expert opinions, industry adoption across chips, cloud services, and applications, and discusses future directions for model scaling, distillation, and AI competition.
AI competitionAI industryDeepSeek
0 likes · 13 min read

HighService, Alibaba’s Pythonic AI service framework, accelerates large‑model inference and maximizes GPU utilization by separating CPU‑GPU processes, offering out‑of‑the‑box quantization, parallelism and caching, and dynamically reallocating idle GPUs across clusters through a master‑worker scheduler to keep online latency low while boosting offline throughput for diffusion and LLM workloads.
AI ServicePythondistributed systems
0 likes · 16 min read

This article details how to train a YOLOv8‑pose model for license‑plate detection, convert it to TensorRT engine, implement C++ inference and preprocessing, expose the functionality via CGO to Go, and assemble a lightweight web service for real‑time plate recognition.
C++GoTensorRT
0 likes · 12 min read


HelloTechAug 9, 2023 · Artificial Intelligence
Device intelligence brings on-device reasoning and real-time inference to smartphones and IoT gateways, delivering low-latency, privacy-preserving, personalized services such as AR/VR enhancements and recommendation re-ranking, while confronting challenges of hardware fragmentation and model size, and complementing cloud AI through architectures like Hala’s MNN-based pipeline.
Edge ComputingReal-time Decisiondevice intelligence
0 likes · 10 min read

FastDeploy is an open‑source toolkit that standardizes AI model APIs and enables developers to deploy vision, NLP, and speech models on diverse hardware—including GPUs, CPUs, Jetson, ARM, and various NPUs—using just three lines of code or a single command, while delivering end‑to‑end performance optimizations.
AI DeploymentCPUEdge Computing
0 likes · 11 min read

By applying high‑performance operators, quantization, and AI compiler optimizations with Alibaba Cloud PAI Blade and Intel Xeon back‑ends, the address‑standardization service’s deep‑learning models achieved up to 4.11× faster end‑to‑end inference without sacrificing accuracy, enabling more complex models and lower latency.
address standardizationai-optimizationhigh-performance operators
0 likes · 19 min read

The article details Meituan's engineering journey from small DNNs to hundred‑gigabyte deep learning models for food‑delivery ads, analyzing online latency and offline efficiency challenges and presenting distributed storage, CPU/GPU acceleration, OpenVINO, TensorRT, CodeGen, and data‑pipeline optimizations that dramatically improve throughput, memory usage, and sample‑building speed.
CPU accelerationGPU accelerationOpenVINO
0 likes · 45 min read

XGBoost Serving is an open‑source, high‑performance inference system built on TensorFlow Serving that adds dedicated servables for pure GBDT, GBDT+FM binary‑classification, and GBDT+FM multi‑classification models, providing automatic version lifecycle management, GRPC/HTTP APIs, and up to 50 % latency reduction, now available on GitHub after successful deployment in iQIYI’s recommendation platform.
GBDTServing ArchitectureXGBoost Serving
0 likes · 12 min read
