How to Crawl and Analyze Bilibili Video Comments with Async Python

The author, an experienced web‑scraper, shows how to collect and analyze comments from the Bilibili video "改革春风吹满地", which has millions of views.

Data Download

Open the video page, locate the comment API request in the browser console, and construct the URL pattern

https://api.bilibili.com/x/v2/reply?pn={pn}&type=1&oid=19390801

. The video has 1129 comment pages, which is sufficient for a simple analysis.

base_url = "https://api.bilibili.com/x/v2/reply?pn={pn}&type=1&oid=19390801"
async def fetch(url):
    async with sem:  # concurrency control
        async with aiohttp.ClientSession() as session:  # create session
            with async_timeout.timeout(10):  # wait up to 10 s
                async with session.get(url) as res:
                    data = await res.json()  # get JSON data
                    print(data)
                    await asyncio.sleep(2)  # pause to avoid being blocked
                    await save_data(glom.glom(data, "data.replies"))  # store comments

Required modules (install via pip install):

aiohttp

async_timeout

uvloop (Unix only)

glom

Data Storage

Comments are saved as JSON, so MongoDB is used. The asynchronous driver motor provides a non‑blocking interface.

class MotorBase:
    _db = {}
    _collection = {}

    def __init__(self, loop=None):
        self.motor_uri = ''
        self.loop = loop or asyncio.get_event_loop()

    def client(self, db):
        self.motor_uri = f"mongodb://localhost:27017/{db}"
        return AsyncIOMotorClient(self.motor_uri, io_loop=self.loop)

    def get_db(self, db='weixin_use_data'):
        if db not in self._db:
            self._db[db] = self.client(db)[db]
        return self._db[db]

async def save_data(items):
    mb = MotorBase().get_db('weixin_use_data')  # database name
    for item in items:
        try:
            await mb.bilibili_comments.update_one(
                {'rpid': item.get("rpid")},
                {'$set': item},
                upsert=True)
        except Exception as e:
            print("Data insertion error", e.args, "Item:", item)

Data Parsing

Retrieve stored comments from MongoDB and write the plain text to a file using aiofiles.

async def get_data():
    mb = MotorBase().get_db('weixin_use_data')
    data = mb.bilibili_comments.find()
    return data

async def m2f():
    data = await get_data()
    async for item in data:
        t = item.get("content").get("message").strip()
        async with aiofiles.open(pathlib.Path.cwd().parent / "msg.txt", 'a+') as fs:
            await fs.write(t)

Data Analysis

To visualise the comment content, a word‑cloud is generated using jieba for Chinese segmentation, wordcloud, and matplotlib. Stop‑words are defined to filter out common, meaningless terms.

# -*- coding: utf-8 -*-
import jieba
import matplotlib.pyplot as plt
from wordcloud import WordCloud

# Load all comments
comments = []
with open('msg.txt', mode='r', encoding='utf-8') as f:
    rows = f.readlines()
    for row in rows:
        comments.append(row)

# Segment words
comment_after_split = jieba.cut(str(comments), cut_all=False)
words = ' '.join(comment_after_split)

# Load stop‑words
STOPWORDS = set(map(str.strip, open('stopwords').readlines()))

# Load background image
bg_image = plt.imread('1.jpeg')

wc = WordCloud(width=2024, height=1400, background_color='white', mask=bg_image,
               font_path='msyhbd.ttf', stopwords=STOPWORDS, max_font_size=400,
               random_state=50)
wc.generate_from_text(words)
plt.imshow(wc)
plt.axis('off')
plt.show()
wc.to_file('ggcfcmd.jpg')

Word‑cloud image used as background:

Final word‑cloud result:

The analysis shows that the video’s catchiness and nostalgic references to Zhao Benshan’s classic lines are the main reasons for its viral popularity.

All code referenced in this article is available at https://github.com/muzico425/bilibilianalysis.git .