Is Web Crawling Legal? Key Risks and Compliance Tips for Data Collectors

Guide: Apply web crawling technology legally, cautiously, and in compliance.

1. Anti‑Unfair Competition Law Dimension

Without the target’s authorization, crawling may violate the Robots protocol, which is recognized as a commercial ethic in the internet search industry. Courts have treated the Robots protocol as a binding industry norm, and violating it can be deemed a breach of the Anti‑Unfair Competition Law (Article 2) concerning honesty and commercial morality.

Furthermore, if crawlers bypass technical protection measures to access information that is otherwise restricted, such actions may infringe on trade secrets, potentially violating Article 9 of the same law. Additionally, because crawling can disrupt the target’s network systems, it may also breach Article 12.

2. Copyright Dimension

Articles, images, comments, and databases on the web can be protected works if they possess originality. Copying and disseminating such data through crawling may infringe the copyright holder’s reproduction and network transmission rights.

For example, in the case of Ma v. Certain Internet Technology Company , the defendant used crawler technology to collect entries from a French‑Chinese technical dictionary without paying royalties, leading to a judgment that the defendant must cease infringement, apologize, and compensate damages.

3. Criminal Law & Cybersecurity Law Dimension

From a technical standpoint, crawlers that overload a website can violate the Cybersecurity Law concerning network operation safety. If the crawler involves unauthorized intrusion, it may also breach Articles 285 and 286 of the Criminal Law.

When personal information is scraped, it may contravene the Cybersecurity Law’s requirements for lawful collection of personal data and could even constitute a crime of illegal acquisition of computer information system data.

Summary

Data crawling can attract regulatory scrutiny and litigation from competitors. Enterprises should therefore observe the following points:

Avoid crawling data from direct competitors to reduce the risk of anti‑unfair competition lawsuits.

Prefer publicly disclosed data and respect Robots protocols and any explicit prohibitions.

Do not exceed one‑third of the target site’s average daily traffic, as recommended by the draft Data Security Management Measures , to prevent service disruption.

Do not bypass or destroy technical measures that block crawlers.

Immediately suspend crawling if the target site issues a clear stop request.