When a Mistyped Function Wiped a Production Database – Lessons Learned

On October 17, 2020, the founder of Keepthescore accidentally deleted the production database after running a function that was intended only for local development.

Because the service uses a DigitalOcean managed database with daily automatic backups, the team restored the system within 30 minutes, but any scores created between 15:47 and 23:21 CET were permanently lost.

The culprit was a function that hard‑coded the host as “localhost” and connected to the production database, dropping and recreating tables.

def database_model_create():
    """Only works on localhost to prevent catastrophe"""
    database = config.DevelopmentConfig.DB_DATABASE
    user = config.DevelopmentConfig.DB_USERNAME
    password = config.DevelopmentConfig.DB_PASSWORD
    port = config.DevelopmentConfig.DB_PORT
    local_db = PostgresqlDatabase(database=database, user=user, password=password, host='localhost', port=port)
    local_db.drop_tables([Game, Player, Round, Score, Order])
    local_db.create_tables([Game, Player, Round, Score, Order])
    print('Initialized the local database.')

The incident highlights the danger of having destructive operations in code that can be executed against production, the necessity of separating credentials for dev and prod, and the critical importance of reliable backups.

Keepthescore now treats any function that can delete a database as highly dangerous and relies on DigitalOcean’s backup system to ensure quick recovery.