How to Build a High‑Performance Magnet Link Crawler with Python and DHT

Magnet Links Overview

Magnet links are peer‑to‑peer identifiers that use metadata rather than a location URL. A typical magnet link looks like magnet:?xt=urn:btih:E7FC73D9E20697C6C440203F5884EF52F9E4BD28 and includes parameters such as:

magnet: protocol name.

xt: exact topic (e.g., BTIH hash) that uniquely identifies the resource.

dn: display name shown to the user.

tr: tracker URL.

kt: keyword for broader searches.

mt: link to a meta‑file containing additional magnet links.

DHT and BitTorrent Seeds

BitTorrent seeds store file metadata and use a Distributed Hash Table (DHT) to locate peers without a central tracker. Each peer runs a DHT node (UDP) that exchanges contact information, enabling decentralized file discovery.

Project Motivation

The author needed a self‑contained tool to fetch magnet links, discovered existing code (simDHT) lacked Python 3 support and documentation, and therefore created magnet-dht, a Python project that crawls magnet links from the DHT network.

Core Implementation

crawler.py joins the DHT network by impersonating a node, contacts large trackers, and gathers magnet links. Results are stored in Redis sets for deduplication. The crawler uses multithreading/multiprocessing to achieve high throughput (≈1 000 000 links in 4 hours on a modest laptop).

$ redis-cli scard magnets

(integer) 1137627

Additional scripts include:

magnet_to_torrent_aria2c.py – converts a magnet link to a .torrent file using aria2.

parse_torrent.py – decodes .torrent files with bencoder to list contained resources.

database.py – wrapper for Redis operations.

utils.py – assorted helper functions.

Usage Instructions

Clone the repository, install dependencies, and run the manager script with appropriate flags.

$ git clone https://github.com/chenjiandongx/magnet-dht.git

$ cd magnet-dht

$ pip install -r requirements.txt  # ensure Redis is installed

# Show help

$ python manage.py -h

# Start the crawler

$ python manage.py -s

# Convert magnet to torrent

$ python manage.py -m

# Parse torrent files

$ python manage.py -p

Key Takeaways

The project demonstrates how to interact with the BitTorrent DHT protocol using Python, leverage Redis for fast de‑duplication, and combine multithreading with external tools like aria2 to build a practical magnet‑link harvesting system.