Unlock Python’s Metaclass Magic: Build Dynamic ORM and Proxy Crawlers

Understanding Python Metaclasses: From “Dao” to Dynamic ORM and Proxy Crawlers

Although many Python developers claim that metaclasses are a feature only 1% of programmers will ever need, every Python object is ultimately created by the built‑in type – the "Dao" of the language. The article uses the Chinese philosophical sequence "道生一，二生三，三生万物" (Dao creates One, One creates Two, Two creates Three, Three creates All) to explain the relationship between type, metaclass, class and instance.

Basic Example – Hello Class

First a simple class is defined in the usual way:

class Hello(object):
    def say_hello(self, name='world'):
        print('Hello, %s.' % name)

hello = Hello()
hello.say_hello()

The output is:

Hello, world.

Using type directly we can create the same class dynamically:

Hello = type('Hello', (object,), dict(say_hello=lambda self, name='world': print('Hello, %s.' % name)))
hello = Hello()
hello.say_hello()

This demonstrates that type (the Dao) can generate both the metaclass and the class itself.

Creating a Metaclass for Automatic Methods

A custom metaclass can inject methods automatically. The example below defines a metaclass that creates a greeting method based on the class name:

class SayMetaClass(type):
    def __new__(cls, name, bases, attrs):
        attrs['say_' + name.lower()] = lambda self, value, saying=name: print(saying + ',' + value + '!')
        return type.__new__(cls, name, bases, attrs)

class Hello(object, metaclass=SayMetaClass):
    pass

hello = Hello()
hello.say_hello('world!')

The output is:

Hello, world! ----调用结果完全一样。

Another Metaclass Example – ListMetaclass

class ListMetaclass(type):
    def __new__(cls, name, bases, attrs):
        attrs['add'] = lambda self, value: self.append(value)
        return type.__new__(cls, name, bases, attrs)

class MyList(list, metaclass=ListMetaclass):
    pass

L = MyList()
L.add(1)
print(L)  # [1]

Building an ORM with Metaclasses

The tutorial then shows how to build a tiny ORM similar to Django’s model system.

class Field(object):
    def __init__(self, name, column_type):
        self.name = name
        self.column_type = column_type
    def __str__(self):
        return '<%s:%s>' % (self.__class__.__name__, self.name)

class StringField(Field):
    def __init__(self, name):
        super(StringField, self).__init__(name, 'varchar(100)')

class IntegerField(Field):
    def __init__(self, name):
        super(IntegerField, self).__init__(name, 'bigint')

class ModelMetaclass(type):
    def __new__(cls, name, bases, attrs):
        if name == 'Model':
            return type.__new__(cls, name, bases, attrs)
        print('Found model: %s' % name)
        mappings = {}
        for k, v in attrs.items():
            if isinstance(v, Field):
                print('Found mapping: %s ==> %s' % (k, v))
                mappings[k] = v
        for k in mappings.keys():
            attrs.pop(k)
        attrs['__mappings__'] = mappings
        attrs['__table__'] = name
        return type.__new__(cls, name, bases, attrs)

class Model(dict, metaclass=ModelMetaclass):
    def __init__(self, **kw):
        super(Model, self).__init__(**kw)
    def __getattr__(self, key):
        try:
            return self[key]
        except KeyError:
            raise AttributeError("'Model' object has no attribute '%s'" % key)
    def __setattr__(self, key, value):
        self[key] = value
    def save(self):
        fields = []
        args = []
        for k, v in self.__mappings__.items():
            fields.append(v.name)
            args.append(getattr(self, k, None))
        sql = 'insert into %s (%s) values (%s)' % (
            self.__table__,
            ','.join(fields),
            ','.join(['?' for _ in args])
        )
        print('SQL: %s' % sql)
        print('ARGS: %s' % args)

Defining a concrete model:

class User(Model):
    id = IntegerField('id')
    name = StringField('username')
    email = StringField('email')
    password = StringField('password')

u = User(id=12345, name='Batman', email='[email protected]', password='iamback')
u.save()

Running the code prints the discovered mappings and the generated SQL statement:

Found model: User Found mapping: name ==> Found mapping: password ==> Found mapping: id ==> Found mapping: email ==> SQL: insert into User (username,password,id,email) values (?,?,?,?) ARGS: ['Batman', 'iamback', 12345, '[email protected]']

Web Page Fetching Helper

import requests

base_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36',
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'zh-CN,zh;q=0.8'
}

def get_page(url):
    headers = dict(base_headers)
    print('Getting', url)
    try:
        r = requests.get(url, headers=headers)
        print('Getting result', url, r.status_code)
        if r.status_code == 200:
            return r.text
    except requests.ConnectionError:
        print('Crawling Failed', url)
    return None

Metaclass‑Based Proxy Crawler

A metaclass collects all crawling methods that start with crawl_ and records their names and callables.

class ProxyMetaclass(type):
    def __new__(cls, name, bases, attrs):
        count = 0
        attrs['__CrawlFunc__'] = []
        attrs['__CrawlName__'] = []
        for k, v in attrs.items():
            if k.startswith('crawl_'):
                attrs['__CrawlName__'].append(k)
                attrs['__CrawlFunc__'].append(v)
                count += 1
        attrs['__CrawlFuncCount__'] = count
        for k in list(attrs.keys()):
            if k.startswith('crawl_'):
                attrs.pop(k)
        return type.__new__(cls, name, bases, attrs)

The ProxyGetter class uses the collected functions to fetch raw proxy lists from different sites.

class ProxyGetter(object, metaclass=ProxyMetaclass):
    def get_raw_proxies(self, site):
        proxies = []
        print('Site', site)
        for func in self.__CrawlFunc__:
            if func.__name__ == site:
                this_page_proxies = func(self)
                for proxy in this_page_proxies:
                    print('Getting', proxy, 'from', site)
                    proxies.append(proxy)
        return proxies

    def crawl_daili66(self, page_count=4):
        start_url = 'http://www.66ip.cn/{}.html'
        urls = [start_url.format(page) for page in range(1, page_count + 1)]
        for url in urls:
            print('Crawling', url)
            html = get_page(url)
            if html:
                doc = pq(html)
                trs = doc('.containerbox table tr:gt(0)').items()
                for tr in trs:
                    ip = tr.find('td:nth-child(1)').text()
                    port = tr.find('td:nth-child(2)').text()
                    yield ':'.join([ip, port])

    def crawl_proxy360(self):
        start_url = 'http://www.proxy360.cn/Region/China'
        print('Crawling', start_url)
        html = get_page(start_url)
        if html:
            doc = pq(html)
            lines = doc('div[name="list_proxy_ip"]').items()
            for line in lines:
                ip = line.find('.tbBottomLine:nth-child(1)').text()
                port = line.find('.tbBottomLine:nth-child(2)').text()
                print(ip + ':' + port)
                yield ip + ':' + port

    def crawl_goubanjia(self):
        start_url = 'http://www.goubanjia.com/free/gngn/index.shtml'
        html = get_page(start_url)
        if html:
            doc = pq(html)
            tds = doc('td.ip').items()
            for td in tds:
                td.find('p').remove()
                yield td.text().replace(' ', '')

Running the crawler:

if __name__ == '__main__':
    crawler = ProxyGetter()
    print(crawler.__CrawlName__)
    for site_label in range(crawler.__CrawlFuncCount__):
        site = crawler.__CrawlName__[site_label]
        myProxies = crawler.get_raw_proxies(site)
        # process myProxies as needed

The metaclass automatically gathers the three crawl_* methods, counts them, and removes them from the class namespace, leaving only the lists __CrawlFunc__ and __CrawlName__ for runtime use – a pattern very similar to the ORM example above.

Conclusion

By treating type as the fundamental "Dao", we can let Python generate classes, inject methods, and build higher‑level abstractions such as an ORM layer or a flexible proxy‑crawling framework. The same metaclass technique that powers the ORM also powers the batch proxy crawler, illustrating the power of "道生一，二生三，三生万物" in real code.