Are Hackers Already Weaponizing AI? Inside the ZeroFOX Experiment

AI‑driven phishing experiment

In a 2022 study conducted by ZeroFOX data scientists, the effectiveness of an AI system (named SNAP_R) was compared with human operators for delivering phishing tweets on Twitter.

Methodology

Target pool: more than 800 Twitter accounts selected as potential victims.

AI approach: SNAP_R generated phishing messages autonomously, posting at a rate of 6.75 tweets per minute.

Human approach: participants manually crafted and posted phishing tweets at 1.075 tweets per minute.

Success metric: number of users who clicked the malicious link.

Results

AI: 6.75 tweets/min → 275 compromised users out of the target pool.

Humans: 1.075 tweets/min → 49 compromised users from 129 attempts.

The experiment demonstrates that AI can scale phishing campaigns by an order of magnitude while achieving a higher compromise rate.

Industry perception of AI‑enabled attacks

At the Black Hat security conference, a survey of attendees asked whether they expect hackers to employ AI within the next year. Sixty‑two percent answered affirmatively, indicating a strong consensus that AI will become a mainstream tool for malicious actors.

Expert commentary

Brian Wallace, senior scientist at Cylance, explained that AI and machine learning address the “scale problem” for attackers: they enable automated selection of targets, timing, and payload generation while minimizing the attacker’s exposure.

Implications

The findings suggest that defensive strategies must account for high‑throughput, AI‑generated phishing content, and that detection systems need to operate at comparable speed and adaptability.