Overview of a Web Page Content Extraction Algorithm and Its Practical Demo

The article presents an algorithm that can automatically parse a web page’s raw HTML to extract structured information such as the article title, publication time, main content, author, and source without needing XPath rules.

A public demo is provided where users can copy a news page’s source HTML, encode it in Base64 using an online tool, paste the encoded string into the demo’s input area, and click the analysis button to view the extracted results, which include interface information, timing statistics, and the parsed fields.

The algorithm’s usefulness is illustrated by comparing it to traditional browser reading modes, describing how it can replace labor‑intensive XPath‑based crawlers in large‑scale news or tender data collection, and highlighting the massive manual effort saved.

Several existing solutions are listed, including 360 Browser’s reading mode, Microsoft’s API, the open‑source Python libraries Readability and GNE, and various academic papers, with a brief evaluation of their efficiency, extraction capability, and accuracy.

Comparative analysis shows that modern GNE, which combines visual heuristics with news‑page feature rules, yields the best extraction results, though it still requires browser rendering and may have performance trade‑offs.

The article concludes that the algorithm can be extended to other domains such as tender information, e‑commerce, and pharmaceutical pages, and that while deep‑learning approaches exist, rule‑based methods like the one described remain competitive.

References: [1] Demo page: http://39.105.152.125:3597/ [2] Example news article: http://glhd.gxnews.com.cn/staticpages/20220321/newgx62388054-20687881.shtml [3] Online Base64 tool: https://www.qqxiuzi.cn/bianma/base64.htm