When Does Programming Cross the Legal Line? A Developer's Risk Guide

Crawler

Web crawlers are a widely used data collection technique that originated in search engines and now support many commercial models such as news aggregation, price comparison, and weather forecasting.

Because crawlers provide great convenience, many companies have dedicated crawler teams and post "crawler engineer" job ads.

Legally, a crawler is just a tool, similar to a kitchen knife, so some argue that "the tool itself is innocent." However, case law shows that developers and users often justify their actions with claims like "I only accessed public data," "I followed robots.txt," or "I used it for personal research without commercial intent."

Is Crawling Illegal?

The main legal reference is the draft "Data Security Management Measures" (May 28, 2019) which covers data collection, processing, and security supervision. Improper use of crawlers can lead to criminal liability, with penalties varying by circumstance.

Whether a crawler is illegal depends on three factors: the type of data collected, the method of collection, and the subsequent use of the data.

What Data Makes Crawling Illegal?

1. Copyright‑protected works

Articles, comments, and other content on websites are protected by copyright. Browsing is legal, but copying and selling such works without permission violates the Copyright Law (Article 46) and can result in civil and administrative penalties.

If crawled data is stored, distributed, or used for profit, it constitutes a crime.

2. Personal information or privacy

Even publicly posted personal data (e.g., on Weibo or WeChat) cannot be harvested arbitrarily. The Civil Code (Article 111) and the Cybersecurity Law (Article 44) prohibit illegal collection, use, processing, or transmission of personal information.

Mass collection of personal data without consent is illegal.

3. Data protected by anti‑unfair‑competition law

User‑generated content that forms a website’s core competitive advantage (e.g., reviews on Dianping or Ctrip) is protected. Unauthorized scraping violates the Anti‑Unfair Competition Law and can be deemed dishonest competition.

How Can Crawling Become Criminal?

Violating robots.txt, bypassing anti‑scraping measures, or forcibly accessing data can breach the Criminal Law (Article 285) and result in imprisonment and fines.

When Is Using Crawled Data Illegal?

Even if the crawler respects robots.txt, using the data for profit, harming others, or engaging in fraud can still be illegal.

Providing personal data to third parties without consent may also violate Article 253‑1 of the Criminal Law.

Gambling and Adult Sites

Developing or supporting gambling or adult websites is a crime if the developer knows the purpose. Providing services worth more than ¥20,000 (e.g., hosting, payment processing, advertising) makes the developer a co‑offender under the Criminal Law (Article 303).

Any involvement in gambling, pornographic, or drug‑related software development is prohibited.

P2P Platforms

Programmers working on illegal P2P lending platforms may be considered accomplices if they are aware of the illegality. If they only receive a regular salary and are unaware of illegal activities, they may not be liable.

Game Cheats (外挂)

Cheat software that modifies game behavior can violate several crimes: illegal business operations, computer system sabotage, and copyright infringement.

Illegal Business Operations

According to Criminal Law Article 225, selling or distributing cheat software for profit can lead to imprisonment and fines.

Computer System Sabotage

Altering game data or disrupting game servers may constitute the crime of damaging computer information systems (Article 286).

Copyright Infringement

Copying and distributing copyrighted game code for profit violates the Copyright Law (Article 217) and can result in imprisonment and fines.

Summary

Developers must avoid crawling personal or copyrighted data without permission, respect robots.txt, and not bypass anti‑scraping mechanisms. Companies that scrape protected content for profit or unfair competition also face legal risks.

Programmers should stay away from gambling, adult, and P2P platforms that operate illegally, and refuse to develop cheat software.

If you work in any of these risky areas, seek legal counsel promptly to assess exposure and protect yourself.