How to Scrape Meituan Waimai App Comments via AJAX and JavaScript

Introduction

A friend needed Meituan Waimai app comment data for a summer project. At first it seemed enough to fetch the page source, but the comments are loaded asynchronously via JavaScript, requiring a different approach similar to previous AJAX scraping cases.

Process

Site analysis Open the Meituan Waimai comment page in a browser and press F12. The first attempt to grab the HTML fails because the comments are loaded via AJAX. By selecting the JS filter in the network panel and checking Preserve log , you can see the AJAX request that returns the comment data. When clicking “load more”, a new request appears with a start parameter that increments by 10 each time.

Inspect the request URL: http://comment.mobilem.360.cn/comment/getComments?callback=jQuery17203361018749253357_1503362214558&baike=%E7%BE%8E%E5%9B%A2%E5%A4%96%E5%8D%96+Android_com.sankuai.meituan.takeoutnew&c=message&a=getmessage&start=0&count=10&_=1503362215647 Changing the start value to i*10 (where i increments) retrieves successive batches of ten comments.

Parse the JSON response from the source code using regular expressions (or a JSON parser) to extract the desired fields.

Implement a loop to request each page, collect the data, and save it to a text file or database.

Code

The actual Python code (shown in the original images) performs the HTTP GET requests, iterates over the start parameter, parses the JSON, and writes the results. (Code omitted here for brevity.)

Result

The final output consists of a structured view of the comments and accompanying files containing the raw data.

Conclusion

Errors are normal; troubleshoot yourself before asking others.

Strengthen knowledge of data type handling.

Thanks to mentors for guidance.