How to Build a Python Crawler to Grab TV Drama Links Automatically

The author, a fan of foreign TV series, wanted to simplify downloading episodes from a site that requires manual browsing. Using Python, they built a web crawler to automatically generate article URLs, check their status, extract download links, and save them to a text file.

Because the site’s article URLs follow the pattern http://cn163.net/archives/<id>/, the crawler iterates over a range of IDs, skips those returning 404, and processes the rest.

Requests is used to fetch each page; the response status code determines whether to continue. The page content is searched with regular expressions to find ed2k links and the episode title.

Extracted links are stored in a dictionary keyed by season and episode numbers, then written to files whose names are sanitized to avoid illegal characters.

Multithreading is employed to run the URL‑generation function in a separate thread, reducing total runtime to about 20 minutes for over twenty thousand entries.

<ol><li><code>def get_urls(self):</code></li><li><code>    try:</code></li><li><code>        for i in range(2015, 25000):</code></li><li><code>            base_url = 'http://cn163.net/archives/'</code></li><li><code>            url = base_url + str(i) + '/'</code></li><li><code>            if requests.get(url).status_code == 404:</code></li><li><code>                continue</code></li><li><code>            else:</code></li><li><code>                self.save_links(url)</code></li><li><code>    except Exception as e:</code></li><li><code>        pass</code></li></ol>

<ol><li><code># -*- coding:utf-8 -*-</code></li><li><code>import requests</code></li><li><code>import re</code></li><li><code>import sys</code></li><li><code>import threading</code></li><li><code>import time</code></li><li><code>reload(sys)</code></li><li><code>sys.setdefaultencoding('utf-8')</code></li><li><code>class Archives(object):</code></li><li><code>    def save_links(self, url):</code></li><li><code>        try:</code></li><li><code>            data = requests.get(url, timeout=3)</code></li><li><code>            content = data.text</code></li><li><code>            link_pat = '"(ed2k://\|file\|[^"]+?\.(S\d+)(E\d+)[^"]+?1024X\d{3}[^"]+?)"'</code></li><li><code>            name_pat = re.compile(r'<h2 class="entry_title">(.*?)</h2>', re.S)</code></li><li><code>            links = set(re.findall(link_pat, content))</code></li><li><code>            name = re.findall(name_pat, content)</code></li><li><code>            # ... processing and writing to file ...</code></li><li><code>        except Exception as e:</code></li><li><code>            pass</code></li><li><code>    def get_urls(self):</code></li><li><code>        # same as above, omitted for brevity</code></li><li><code>    def main(self):</code></li><li><code>        thread1 = threading.Thread(target=self.get_urls)</code></li><li><code>        thread1.start()</code></li><li><code>        thread1.join()</code></li><li><code>if __name__ == '__main__':</code></li><li><code>    start = time.time()</code></li><li><code>    a = Archives()</code></li><li><code>    a.main()</code></li><li><code>    end = time.time()</code></li><li><code>    print(end - start)</code></li></ol>

The author notes that multithreading offers limited benefit due to Python’s GIL, and that handling illegal characters in filenames was a major obstacle.