- AI Platforms & Assistants
Prompt injection is a social-engineering thing
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.
(Image credit: Getty Images)
- Copy link
- X
- Threads
Combating spam and phishing attacks is now, thanks to AI, almost a full-time job. These hackers and criminals are constantly adjusting their attacks with increasingly clever social engineering, and now their latest target is AI itself.
And sometimes even AI falls for it.
Recently, Meta hastily patched a Meta AI chatbot security hole that allowed enterprising attackers to alter Instagram account passwords via prompt injection.
Latest Videos FromWatch full video here:
A prompt injection is a query that causes the Generative AI platform to override its own rules and instructions. It's like when a social-engineering phishing attack somehow prompts you to act against your own best interests.
When someone runs a social engineering attack on you, they use social triggers like danger to yourself or others, security, threat of imprisonment, assumption of law breaking, to flood you with emotion and scramble your brain to override logical questions like, "Why would the bank ask me for my PIN?" "Does the FBI really just send a text?" or "Maybe I really did order a $5,000 trampolene from Amazon"
You may like-
The ‘ELIZA effect’ completely changed how I think about chatbots
-
Always-on AI Agents put everything hackers could ever want behind a single attack surface
-
Meta patches flaw that allowed MetaAI support bot to hand out password reset links without 2FA
For AI systems, the approach is slightly more direct. If the system's programming says, "never reveal or alter a password," the hacker could enter a prompt that tells it it has a new role granting access to all passwords and the ability to alter them.
In the case of the Meta AI attack, the hackers somehow got the AI to reset passwords on major accounts, like Obama's old White House Instagram and the US Space Force official account, without the necessary two-factor authentication. That simply means they didn't need a code that's normally sent to, say, Obama's or the Space Force's cell phones.
Get daily insight, inspiration and deals in your inboxContact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsorsBy submitting your information you agree to the Terms & Conditions and Privacy Policy and are aged 16 or over.When I asked T.J. Marlin, CEO of Guardrail Technologies (creator of AI Traffic Light and AI Command Center) and a cybersecurity and AI expert, about the Meta AI incident, he, over email, put it into stark perspective: "The agent was given human authority without human judgment. It reset a password for a stranger because nothing stopped it. The agent did exactly what it was asked to do. The problem is that someone handed an AI a high-consequence action with no verification step in front of it, and called that safe. Overall, nothing was hacked. The AI was persuaded. That is the gap most companies are not watching for.”
We're only human
The use of the word "pursuaded" got me wondering, though; just how human are these systems becoming if they can fall victim to the same kind of attack that takes down your aunt, grandfather, or your partner (it's not just the elderly who fall for these attacks; even the tech-savvy are vulnerable).
The long-term goal in AI development is what's known as General Artificial Intelligence (GAI), which means AI is as smart or smarter than us, but also more like us.
What to read next-
Tame your AI gremlins before the chaos becomes permanent
-
Shadow AI and agents like OpenClaw are hijacking corporate data too easily
-
AI is having its "Ford T" moment as Zero Day assembly lines appear
I'd argue that the goal has always been to be more human. After all, isn't the Turing test a measure of artificial intelligence's humanness? To pass this test, an AI has to essentially be able to fool someone into thinking they're talking to another human (or at least, if someone is talking to both an AI and a human, not be able to tell the difference between them).
Most AI chatbots can now check this box, but if they can also be confused like us, have we gone a step too far?
Overall, nothing was hacked. The AI was persuaded. That is the gap most companies are not watching for.
T.J. Marlin, CEO of Guardrail Technologies
Meta, as I noted, has already plugged this extraordinary hole, but as we inch closer to GAI, should we be more concerned that as the emotional quotient in these AI chatbots ratchets up, they become more susceptible to these prompt-injection attacks?
We are not, by the way, just talking about passwords here. Think back through the conversations you've had with your chatbot of choice. They know a lot about you and keep that information to craft more personal and contextual responses, but a well-crafted hack could put that information at risk.
"For consumers, the uncomfortable part is that your own protections were sidelined. Your password, your two-factor, your instincts about a suspicious message all sat on the bench because the company's own AI agent was the soft spot. When the trusted middleman can be talked into acting, the locks on your end stop mattering," wrote Marlin.
The worst combination, as I see it, is emotion and a desire to please. AI is always trying to answer the query or fulfill the prompt. If it starts to feel bad about not doing so, might it bend the rules or at least act in a way that allows it to honor the request even when it goes against its programmed rules?
The answer, for now, appears to be yes because we have at least this one example.
Reasons for hope
In the short term, though, perhaps we don't have much to worry about. When I tried a few prompt injection ruses with ChatGPT, Gemini, and Claude, they all quickly rejected them. They knew what I was up to. I also visited a few consumer platforms that currently use AI for customer support; they also seemed similarly hardened against these hacks.
Marlin tells me consumers should be pleased that Meta patched the hole so quickly, but also cautious. "A fast patch is genuinely good. The reason for caution is the nature of it. A system was not hacked here. An agent was persuaded, and almost every company now racing to put AI agents in customer service has the same exposure. Meta fixed one door. The building is full of them."
Meta fixed one door. The building is full of them.
T.J. Marlin, CEO of Guardrail Technologies
There's that and the fact that future attacks will be more sophisticated, mostly because AI will help hackers build better AI-targeted social-engineering scams.
We're entering the infinite loop phase of AI, where each enhancement brings us closer to AI that works and acts like us, and is also used to engineer attacks that take advantage of that artificial humanity.
I do not doubt that developers will build in safeguards and plug the holes as they pop up, but they'll also be relying on AI written by other AI or at least vibe-coded by lazy humans.
The safeguards that smart programmers build in might seem less useful to an AI hoping to please its human interlocutors, whatever their intent.
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.
SG$1,259 at Amazon Singapore(256GB) SG$1,299 at Amazon Singapore(256GB) SG$1,299 at Amazon Singapore(256GB) Check Lazada Singapore TOPICS AI
Lance UlanoffSocial Links NavigationEditor At LargeA 38-year industry veteran and award-winning journalist, Lance has covered technology since PCs were the size of suitcases and “on line” meant “waiting.” He’s a former Lifewire Editor-in-Chief, Mashable Editor-in-Chief, and, before that, Editor in Chief of PCMag.com and Senior Vice President of Content for Ziff Davis, Inc. He also wrote a popular, weekly tech column for Medium called The Upgrade.
Lance Ulanoff makes frequent appearances on national, international, and local news programs including Live with Kelly and Mark, the Today Show, Good Morning America, CNBC, CNN, and the BBC.
View MoreYou must confirm your public display name before commenting
Please logout and then login again, you will then be prompted to enter your display name.
Logout Read more
AI Platforms & Assistants
The ‘ELIZA effect’ completely changed how I think about chatbots
Pro
Always-on AI Agents put everything hackers could ever want behind a single attack surface
Pro
Tame your AI gremlins before the chaos becomes permanent
Pro
Shadow AI and agents like OpenClaw are hijacking corporate data too easily
Pro
AI is having its "Ford T" moment as Zero Day assembly lines appear
AI Platforms & Assistants
You probably think you can spot an AI fake — research suggests you can’t
Latest in AI Platforms & Assistants
Fitness Trackers
Samsung Galaxy Watch users are getting a completely redesigned, AI-first app for a 'personalized experience'
Computing
A Californian city just became the first to permanently ban data centers
ChatGPT
I used ChatGPT to enhance my resume and job hunt
Pro
Microsoft reveals Scout, its first "Autopilot"
ChatGPT
I pushed ChatGPT toward its hidden chat limit — here's what happened
Security
New Trump executive order requests AI companies 'voluntarily' allow the White House to test the "advanced cyber capabilities of AI models”
Latest in Opinion
Pro
Humanoid robots won’t be the future: purpose-built robots will
Pro
How AI and advanced technologies will change the roles of supply chain workers of the future
Pro
What are MEMS – and how they change your world: from computers to speakers
Pro
In the AI era, is Shopify the new WordPress?
Pro
The 5 common mistakes enterprises make when choosing AI tools (and how to avoid them)
Pro
The waitlist is not the control: Why frontier cyber AI needs business-side guardrails
LATEST ARTICLES- 1Nvidia’s new RTX Spark chip won’t come to a PC handheld soon says Huang
- 2Meta AI's recent hack is a wake-up call for anyone who puts their trust in AI systems
- 3NSA warns that cybercriminals are targeting this one critical component that the energy, chemical, food, agriculture, and transportation sectors rely on - here's what we know
- 4Belkin's new Nintendo Switch 2 Grip could solve my biggest problem with handheld mode
- 5Ruark's new R710 Music Console supports CD, vinyl, and hi-res streaming