Why My Data‑Generation Job Crashed the Server: A Real‑World Spring Boot Debugging Tale

1. Overview

A scheduled job writes fake data into a database. It must handle both historical dates (passed in) and real‑time execution (triggered every ten seconds). Some records are based on real data scraped from the internet, so random generation must reuse existing rows.

2. Program Logic

2.1 Job Interface Definition

/**
 * desc:
 * 造数据的job，可按表来划分。一个表一个job
 */
public interface DataProduceJob {
    /** job initialization */
    void jobInit(Date date);
    /** job execution details */
    void jobDetail(Integer recordNum);
}

The original design split initialization and execution into two methods, which later got merged.

2.2 Faulty Job Implementation

@Component
public class TopicWebsiteJob extends BaseJob implements DataProduceJob {
    @Autowired
    private TopicWebsiteMapper topicWebsiteMapper;
    private Date date;
    Random random = new Random();
    private List<TopicWebsite> topicWebsites;

    @Override
    public void jobInit(Date date) {
        this.date = date;
        topicWebsites = topicWebsiteMapper.selectAll();
    }

    @Override
    public void jobDetail() {
        for (TopicWebsite website : topicWebsites) {
            for (int i = 0; i < 5; i++) {
                TopicWebsite topicWebsite = new TopicWebsite();
                topicWebsite.setWebsiteName(website.getWebsiteName());
                topicWebsite.setIconUrl(website.getIconUrl());
                topicWebsite.setHotValue((long) random.nextInt(6354147));
                topicWebsite.setCreateTime(date);
                topicWebsiteMapper.insert(topicWebsite);
            }
        }
    }
}

The job reads all existing rows during initialization and then creates new rows based on them.

2.3 History Data Initializer

public interface Initer {
    /** initialization logic */
    Boolean init();
}

@Component
public class TopicWebsiteIniter implements Initer {
    @Autowired
    private TopicWebsiteJob job;

    @Override
    public Boolean init() {
        DateTime now = DateTime.now();
        for (int a = -29; a < 1; a++) {
            for (int b = 0; b < 24; b++) {
                int minutes = (int) (Math.random() * 60);
                Date date = DateUtils.getNeedTime(b, minutes, 0, a);
                if (a == 0 && date.after(now.toDate())) {
                    // skip future times
                } else {
                    job.jobInit(date);
                    job.jobDetail(360);
                }
            }
        }
        return true;
    }
}

The initializer generates historical data by creating a job for each past hour and minute within the last 30 days.

2.4 Scheduler

@Component
public class TopicWebsiteScheduler implements DataProduceScheduler {
    private static final Logger logger = LoggerFactory.getLogger(TopicWebsiteScheduler.class);
    @Autowired
    private TopicWebsiteJob job;

    @Override
    @Scheduled(cron = "0/10 * * * * ?")
    public Boolean schedule() {
        logger.info("start...");
        job.jobInit(new Date());
        job.jobDetail(1);
        return true;
    }
}

The scheduler triggers the job every ten seconds in production.

3. Problem Manifestation

When the scheduler runs, jobInit is called repeatedly, each time loading **all** rows from the table. Because the job also inserts the generated rows back into the same table, the row count doubles on each trigger (10 → 20 → 40 → 80 …). This exponential growth quickly inflates the table to millions of rows, exhausting JVM memory and causing the application to hang.

4. Conclusion

The root cause was a design oversight: using the mutable table as both source and destination for fake data without limiting the read set. Proper separation of source data, batch limits, or a distinct staging table would prevent the exponential blow‑up and keep the job memory‑safe.