People have been worrying about fake news for years, but generative AI has added a new wrinkle. It is now trivial for almost anyone to produce realistic looking text, images, or video that can be used to mislead others. Recent research is starting to show how this AI generated misinformation actually behaves once it hits social media feeds.
In Short
- Researchers collected examples of AI generated misinformation circulating on major social platforms.
- They found that smaller accounts, not big influencers, were responsible for a surprising share of AI based misinformation posts.
- AI generated falsehoods can attract high levels of engagement, sometimes matching or exceeding traditional misinformation.
- Many AI based posts use a relatively positive or neutral emotional tone rather than obvious anger or outrage.
- The same AI tools that help create misinformation could also help detect patterns and flag suspicious content.
How The Study Worked
A 2025 study titled “Characterizing AI Generated Misinformation on Social Media” looked at posts that combined generative AI outputs with misleading or false claims. The authors tracked how those posts spread, who posted them, and how users reacted.
Instead of focusing only on high profile accounts, the researchers paid attention to smaller users who were experimenting with AI tools. By comparing engagement numbers and network patterns, they could see whether AI based misinformation behaved differently from older kinds of fake news.
Smaller Accounts, Big Footprint
One of the more surprising findings was that a large portion of AI generated misinformation was posted by relatively small accounts rather than big, verified profiles. These users may not have huge followings, but AI tools give them the ability to produce professional looking content that can still travel widely through shares and algorithmic recommendations.
This pattern is a challenge for platforms and fact checkers. Moderation systems that focus mainly on prominent accounts or known repeat offenders may miss a long tail of new users who are constantly generating fresh falsehoods with help from AI.
Engagement And Emotional Tone
The study also found that AI generated misinformation can achieve significant engagement, including likes, shares, and comments that rival older types of misleading content. In some cases, it spreads faster because it looks cleaner or more polished than hastily written human posts.
Another twist is emotional tone. While traditional misinformation often leans heavily on anger or fear, many AI generated posts examined in the research used a more positive, calm, or speculative style. That makes them harder to spot using simple “angry language” filters and may help them slip past people’s suspicion.
AI As Part of the Problem and the Toolkit
Other studies and reviews point out that large language models and image generators can flood the internet with low-cost misinformation. At the same time, researchers are exploring ways to use similar technology to detect suspicious patterns, label AI content, and assist human fact checkers.
In practice, there is unlikely to be a single technical fix. The findings suggest that platforms will need a mix of detection tools, user education, and transparency about AI generated media if they want to keep the worst content in check.
In Short
- AI has lowered the barrier to creating convincing fake content, and smaller accounts are using that power in ways that matter.
- AI generated misinformation can spread quickly even when it does not rely on over-the-top emotional language.
- The same technology that creates the problem might end up being part of the solution, but only if platforms and researchers keep pace.
Sources
- ArXiv – “Characterizing AI-Generated Misinformation on Social Media”
- AI & Society – “Generative AI and misinformation: a scoping review of the implications”
- Harvard Misinformation Review – “Misinformation reloaded? Fears about the impact of generative AI are overblown”
- University of Florida – “AI and Misinformation” background explainer