BestHub
Sign in
Tag

DataLoader

0 views collected around this technical thread.

OPPO Kernel Craftsman
OPPO Kernel Craftsman
Mar 29, 2024 · Artificial Intelligence

InternLM Model Research and XTuner Practical Guide (Part 1): DataLoader, Model Conversion, Merging, and Inference

The guide walks through fine‑tuning InternLM‑Chat‑7B with XTuner, showing how to build a DataLoader from a HuggingFace Dataset, convert a LoRA .pth checkpoint to HuggingFace format, merge the adapter into the base model, run inference, and adapt the process for custom datasets and 4‑bit quantization experiments.

DataLoaderFineTuningInternLM
0 likes · 27 min read
InternLM Model Research and XTuner Practical Guide (Part 1): DataLoader, Model Conversion, Merging, and Inference
HomeTech
HomeTech
Oct 12, 2022 · Backend Development

Practical Experience of Introducing GraphQL as a BFF Layer in Backend Development

This article shares a backend team's practical experience of adopting GraphQL with Express and Apollo Server as a BFF layer, covering motivations, implementation steps, schema merging, proxy configuration, HTTP caching strategies, DataLoader optimization, code generation, and integration testing to improve performance and developer efficiency.

Apollo ServerDataLoaderExpress
0 likes · 14 min read
Practical Experience of Introducing GraphQL as a BFF Layer in Backend Development
Python Programming Learning Circle
Python Programming Learning Circle
Oct 11, 2021 · Artificial Intelligence

How PyTorch DataLoader Transfers Data Between Processes Using Multiprocessing Queues

This article explains the internal mechanism of PyTorch's DataLoader when using multiple worker processes, detailing how tensors are serialized, shared via multiprocessing.Queue, and reconstructed in the main process to avoid unnecessary memory copies.

DataLoaderPyTorchSerialization
0 likes · 7 min read
How PyTorch DataLoader Transfers Data Between Processes Using Multiprocessing Queues
Python Programming Learning Circle
Python Programming Learning Circle
Aug 23, 2021 · Artificial Intelligence

Efficient PyTorch Training Pipeline: Tips, Profiling, and Multi‑GPU Strategies

This article presents practical strategies for building high‑performance PyTorch training pipelines, covering bottleneck identification, efficient data loading, RAM‑based datasets, profiling tools, multi‑GPU training with DataParallel and DistributedDataParallel, custom loss implementation, and hardware‑vs‑software trade‑offs to accelerate deep‑learning workloads.

Custom LossDataLoaderMulti-GPU
0 likes · 13 min read
Efficient PyTorch Training Pipeline: Tips, Profiling, and Multi‑GPU Strategies