Why Is ChatGPT Getting Lazier? A Statistical Dive into Seasonal Performance

Background

Since the OpenAI developer event on November 6, many users have reported that GPT‑4 seems to produce shorter, less detailed code responses. OpenAI later acknowledged the issue without providing a concrete cause.

Experiment Design

Rob Lynch used the GPT‑4‑turbo API to create two system prompts: one stating the current month is May, the other stating it is December. For each prompt he asked the model to "complete a machine‑learning‑related coding task" using identical user instructions.

Data Collection

He collected 477 responses for each month and measured the character count of the generated code.

May prompt: average length 4298 characters

December prompt: average length 4086 characters

The difference of about 200 characters was statistically significant (t‑test p < 2.28e‑07).

Community Interpretation

Many participants hypothesized that the model may have learned from its training data that humans slow down in December, effectively giving itself a "holiday". Others suggested the effect could extend to weekends versus weekdays, or be influenced by the model’s temperature setting.

Academic Context

Separate research from Stanford and UC Berkeley has shown that GPT‑4’s compliance with user instructions can vary over time, indicating that model behavior is not static. Tsinghua professor Ma Shaoping also discussed how temperature parameters might affect output length.

Re‑evaluation Attempts

Another user tried to replicate the findings with the ChainForge prompt‑engineering GUI, running 80 samples for each month. Their t‑test did not reach significance (p ≈ 0.089), suggesting that sample size and the metric (character count vs token count) influence the result.

Limitations and Open Questions

The original test cost about $28 per run, limiting larger‑scale sampling. Differences between character‑based and token‑based measurements, as well as the impact of temperature, remain unresolved. Further systematic studies are needed to determine whether the observed seasonal dip is a genuine model property or an artifact of prompt design.

Conclusion

While statistical evidence points to shorter outputs in December, the cause—whether seasonal bias, temperature settings, or other factors—remains uncertain. Ongoing community experiments and academic research are essential to clarify the dynamics of large‑language‑model performance over time.