How to Fix Text Encoding Issues in Python Web Scraping (Step‑by‑Step)

1. Introduction

Hello, I'm PiPi. Recently a member of the Python community asked about garbled text when crawling web pages with Python.

2. Code Example

url = 'https://www.fwsir.com/Article/html/Article_20230101085126_2180223.html'
heades = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'}
r = requests.get(url=url,headers=heades).text
r.encode('utf-8')
soup = BeautifulSoup(r,'html.parser')
s1 = soup.select('div.content>p')
for i in s1:
    print(i.get_text())

3. Solution Process

The encoding problem is common. A community member provided a hint, illustrated with screenshots.

Another member shared a practical screenshot confirming the code works.

A third contributor added additional crawling knowledge.

4. Summary

This article presented a typical Python web‑scraping text‑encoding issue, explained the cause, and provided a complete code solution that successfully extracts readable text.

When asking similar questions, it is recommended to share a small, anonymized data sample, include the error screenshot, and keep the code concise (under 50 lines) or attach a .py file for longer scripts.