Build a Python Crawler to Automatically Grab Drama Download Links

Since the Chinese regulatory restrictions limited direct streaming of foreign TV series, the author discovered a download site called "天天美剧" and decided to automate link collection using a Python crawler.

The crawler generates sequential archive URLs (e.g., http://cn163.net/archives/24016/) within a specified range, skips URLs returning a 404 status, and processes the remaining pages.

For each valid page, the script downloads the HTML, uses a regular expression to find ed2k links of the form ed2k://|file|..., and extracts the season and episode numbers from the link to sort them correctly.

The extracted links are written to a text file named after the drama title (spaces are allowed, but slashes and other illegal characters are removed), with each link on a new line.

def get_urls(self):
    try:
        for i in range(2015, 25000):
            base_url = 'http://cn163.net/archives/'
            url = base_url + str(i) + '/'
            if requests.get(url).status_code == 404:
                continue
            else:
                self.save_links(url)
    except Exception as e:
        pass

# -*- coding:utf-8 -*-
import requests, re, sys, threading, time
reload(sys)
sys.setdefaultencoding('utf-8')

class Archives(object):
    def save_links(self, url):
        try:
            data = requests.get(url, timeout=3)
            content = data.text
            link_pat = "\"(ed2k://\|file\|[^\"]+?\.(S\d+)(E\d+)[^\"]+?1024X\d{3}[^\"]+?)\""
            name_pat = re.compile(r'<h2 class="entry_title">(.*?)</h2>', re.S)
            links = set(re.findall(link_pat, content))
            name = re.findall(name_pat, content)
            links_dict = {}
            for i in links:
                links_dict[int(i[1][1:3]) * 100 + int(i[2][1:3])] = i[0]
            with open(name[0].replace('/', ' ') + '.txt', 'w') as f:
                for i in sorted(links_dict.keys()):
                    f.write(links_dict[i] + '
')
        except Exception as e:
            pass
    def main(self):
        thread1 = threading.Thread(target=self.get_urls())
        thread1.start()
        thread1.join()

if __name__ == '__main__':
    start = time.time()
    a = Archives()
    a.main()
    end = time.time()
    print(end - start)

The full script also employs multithreading, though due to Python's GIL the performance gain is limited; the entire collection of over 20,000 dramas completes in under 20 minutes after filtering out invalid URLs.

A notable challenge was handling file names: while spaces are allowed in .txt filenames, characters such as slashes, backslashes, and parentheses are not, requiring additional sanitization of drama titles.

Source: CodeCEO – "Python Crawling Drama"