How to Scrape Meituan Takeout App Comments with Python and MongoDB

1. Introduction

During a summer internship the author needed Meituan Takeout app comment data. At first they tried to fetch the page source, but discovered the comments are loaded asynchronously via JavaScript (Ajax), so a simple request is insufficient.

2. Process

Steps:

Analyze target site: open Meituan Takeout comments page and press F12.

Identify that comments are loaded via Ajax. In the Network panel, select the JS request that returns comment data, enable "Preserve log", and notice that clicking "load more" triggers a new Ajax request with a changing start parameter that increments by 10.

Inspect the request URL, e.g.

http://comment.mobilem.360.cn/comment/getComments?callback=jQuery...&baike=%E7%BE%8E%E5%9B%A2%E5%A4%96%E5%8D%96+Android_com.sankuai.meituan.takeoutnew&start=0&count=10&_=...

. Varying start as i*10 retrieves subsequent pages.

Parse the JSON response using regular expressions to extract fields such as content and username.

Loop through pages, fetch data, and save to a text file and MongoDB.

3. Code

Configuration (config.py):

MONGO_URL='localhost'
MONGO_DB='meituan'
MONGO_TABLE='meituan'

Main script (Python 3):

import requests
from requests.exceptions import RequestException
import json, re, pymongo

client = pymongo.MongoClient(MONGO_URL)
db = client[MONGO_DB]
base_url = 'http://comment.mobilem.360.cn/comment/getComments?callback=jQuery17209056727722758744_1502991196139&baike=%E7%BE%8E%E5%9B%A2%E5%A4%96%E5%8D%96+Android_com.sankuai.meituan.takeoutnew&start='

def the_url(url):
    try:
        response = requests.get(url)
        if response.status_code == 200:
            response.encoding = 'utf-8'
            return response.text
        return None
    except RequestException:
        print('请求出错')
        return None

def the_total():
    html = the_url(base_url)
    pattern1 = re.compile('"total":(.*?),"messages"', re.S)
    Total = int(':'.join(re.findall(pattern1, html)))
    print(f'总计评论{Total}条')
    write_to_file(f'总计评论{Total}条')
    return Total

def parse_one_page(html):
    pattern2 = re.compile('"m_type":"0",(.*?),"username"', re.S)
    items = re.findall(pattern2, html)
    for item in items:
        item = "{" + item + "}"
        item = json.loads(item)
        write_to_file(item)
        print(item)
        save_to_mongo(item)

def save_to_mongo(result):
    try:
        if db[MONGO_TABLE].insert(result):
            print('储存到MongoDB成功', result)
    except Exception:
        print('储存到MongoDB失败', result)

def write_to_file(content):
    with open('meituan_result.text', 'a', encoding='utf-8') as f:
        f.write(json.dumps(content, ensure_ascii=False) + '
')

def main():
    Total = the_total()
    Total = int(Total/10) + 2
    for i in range(Total):
        url = base_url + str(i*10)
        if the_url(url):
            html = the_url(url)
            parse_one_page(html)
        else:
            print('输完啦')
    ps = 'PS：因为有些评论空，所以实际评论比抓取的少'
    write_to_file(ps)
    print(ps)

if __name__ == '__main__':
    main()

4. Result Data View and Files

(Image showing the saved data file and MongoDB collection.)

5. Summary

1. Program errors are normal; troubleshoot yourself before asking others.

2. Strengthen knowledge of data type handling.

3. Thanks to contributors "皮皮哥" and "姚文峰".