What Happens When Most Users Quit a Vocabulary App? A Data‑Driven Investigation

0x00 Introduction

The author wonders how many users of a popular vocabulary‑learning app actually finish their first day of study and how many persist over the long term.

0x01 Problem Statement and Task Breakdown

Key questions include: how many users keep studying for more than 100 days, how many dreams are lost due to lack of persistence, and whether the amount of words learned follows a normal distribution.

0x02 Task 1 – Data Crawling

Public user data from Shanbay (approximately 1/30 of all users) is scraped using Scrapy on Python 2. Anti‑scraping measures require rotating proxy servers and disabling cookies.

0x03 Task 2 – Data Cleaning and Storage

Extracted data is cleaned with SQL statements and pandas, then stored in a PostgreSQL database.

0x04 Task 3 – Data Analysis

Analysis is performed in an IPython notebook (Python 3 via Anaconda) using pandas, seaborn and other libraries.

0x05 Conclusions

A total of 1,111,111 pages were crawled, yielding 610,888 user records. Notable extremes: longest check‑in streak 1,830 days (user *chainyu*), highest growth value 28,767 (user *Lerystal*), most words learned 38,313 (user *chenmaoboss*). Average per user: 14.18 check‑in days, 121.79 growth value, 78.92 words; only about 11 % exceed average check‑ins or growth, and ~2 % exceed average word count.

Histograms show a heavily skewed distribution of check‑in counts, with most users having zero or very few days. The data is segmented into ranges 0‑20, 20‑100, 100‑500, and 500‑2000 days, each illustrated by the following images:

Retention statistics show that a large majority abandon the app early: 68.15 % stop before day 0, 76.40 % before day 1, 79.31 % before day 2, and only about 2 % persist beyond 100 days.

The analysis concludes that most learners give up quickly, the distribution of study days is far from normal, and the low overall effort makes talent less relevant.

0x06 Reflections on the Process

Scraping limitations include IP bans after excessive requests and the need for reliable proxy servers; disabling cookies is also essential. The author notes missing error handling and the difficulty of choosing appropriate statistical models.

0x07 Code

The full scraping and analysis code is hosted on GitHub (https://github.com/twocucao/DataScience/). The repository contains the Scrapy spider and the IPython notebook used for the analysis.