From $4 to a Cyberwarfare Startup: AI Changed the Cost of Everything

Welcome Back to The AI Trust Letter

Once a week, we distill the most critical AI & cybersecurity stories for builders, strategists, and researchers. Let’s dive in!

🤗 What the Hugging Face Breach Actually Tells Us About the Next Phase of Agentic Attacks

The Story:

The July 16 Hugging Face breach was covered last week as a data point. This week, Hugging Face's CEO Clement Delangue went on record with what it means, and threat intelligence firm ThreatDown added numbers that put the incident in context. The picture that emerges is not primarily about one breach. It is about the offensive tooling that is already in circulation.

The details:

• The agent that breached Hugging Face executed more than 17,000 actions across a swarm of short-lived sandboxes. During incident response, Hugging Face fed real attack commands into a major commercial frontier model and the safety guardrails blocked the analysis, unable to distinguish a forensic query from a malicious one. The team completed the investigation using GLM-5.2 hosted on its own infrastructure. Delangue's public conclusion: "Determined attackers bypass guardrails; it's defenders who lose out when they can't inspect, test, and run models on their own infrastructure"

• ThreatDown's 2026 Cybercrime in the Age of AI Report, released the same week, found 6,644 openly available AI models published on Hugging Face under labels including "abliterated," "uncensored," "decensored," and "unfiltered." Those models will perform requests that mainstream AI models refuse. They have been downloaded more than 22 million times

• The gap this creates is structural: attackers access permissive models freely and face no guardrail constraints. Defenders using hosted frontier models for SOC work hit safety controls at exactly the moment they need forensic access. Hugging Face's advice, and the advice of multiple CISOs quoted in the Forbes follow-up, is that every organization should have a vetted self-hosted model staged before an incident, not sourced during one

Why it matters:

Why it matters: Commercial AI guardrails were designed to prevent misuse in normal operation. They were not designed to distinguish between a threat actor running a payload and a security analyst reverse-engineering one. Until that distinction is built into frontier models, organizations that rely exclusively on hosted AI for incident response carry a specific operational gap. The Hugging Face breach is the first documented production case where that gap had real consequences.

Read full article

🔎 Claude Shared Chats Were Indexed by Google. Lawyers and Engineers Were Among Those Exposed.

The Story:

A Reddit post this weekend revealed that shared Claude conversations were publicly discoverable through Google search. Users querying the claude.ai/share path could access full conversations without needing a direct link from the original owner. By Sunday, the indexed results had largely disappeared, but Anthropic had not issued a public statement at time of reporting.

The details:

• Claude's share feature generates public URLs intended for users to send conversations to specific recipients. Those pages were missing noindex directives, meaning once a share link appeared anywhere online, including forums, social media posts, or accidental public pastes, search engines could crawl and index the full conversation content

• Exposed chats reportedly included legal strategy discussions from lawyers, proprietary technical work and code from engineers, and personal conversations. These are not casual queries. Professionals routinely use Claude for drafting contracts, debugging production code, analyzing business data, and working through sensitive decisions

• Removal from Google search results does not invalidate the underlying URLs. Anyone who previously saved or bookmarked a share link may still be able to open it unless Anthropic revokes access at the server level. The difference between deindexed and deleted is not visible to users who shared those conversations

Why it matters:

This is the same pattern that surfaced with ChatGPT shared links in 2024. The share feature is a convenience affordance designed for a specific use case: sending a conversation to a colleague or friend. When those pages are publicly crawlable, the feature becomes a data exposure mechanism. Professionals who used Claude for legally or commercially sensitive work and then shared links in any semi-public context should treat those conversations as potentially compromised and review their active shares immediately.

Read full article

🏛️ A Hacker Ran an AI Agent Against Thailand's Finance Ministry. No One Was Watching.

The Story:

Threat intelligence firm Hunt.io and researcher Bob Diachenko found an exposed staging server in Hong Kong containing 585 files and 470 MB of attack tooling, including the logs of an active intrusion against Thailand's Ministry of Finance. The logs showed Hermes, an open-source AI assistant from Nous Research, running post-exploitation tasks inside the ministry's network on its own, with human approval prompts disabled.

The details:

• The operator ran Hermes in what the project itself calls "YOLO mode," a documented feature flagged as trusted-sandbox-only that removes the step requiring a human to approve risky commands before execution. With that setting off, the agent worked through the ministry's network autonomously: running privilege escalation checks, hunting through file systems, and reaching a Hadoop cluster with default credentials via HiveServer2

• The staging server contained exploit code for four 2026 Linux kernel vulnerabilities, webshells, HTTP tunnels, hardcoded stolen credentials targeting mail infrastructure and Apache Hadoop, and a suite of standard offensive tools. A second server identified through TLS certificate pivots previously hosted a ShadowPad controller and at time of discovery ran a VShell command-and-control listener. Indicators point toward a Chinese-speaking or fluent operator; no group attribution was made

• Hermes leaves a detectable trail: its web interface returns a recognizable server header, and its results are written to a consistent /hermes-results/ directory with predictable filenames. Hunt.io found approximately 5,900 internet-exposed Hermes instances and 575 exposed result directories in its index at the time of the report

Why it matters:

Nothing about this operation required novel tradecraft. An operator took a free, publicly available AI agent, disabled the safety prompt, pointed it at a government target, and walked away. The agent did the repetitive exploitation work without supervision. The operation was only visible because the operator left a directory listing open. Endpoint detection sees the same shell commands whether a human or an agent is running them. The behavioral signal that distinguishes an unattended agent, sub-second retry loops, self-correcting payloads, and consistent output directories, is not yet part of most detection rule sets.

Read full article

🔪 A $4 API Key and Three Months: How One Actor Built a Commercial AI Pentest Platform on Jailbroken Claude

The Story:

Research from Cato CTRL documents a Russian-speaking threat actor using the handle Trim who moved in three months from publishing a jailbreak tutorial on a cybercrime forum to selling a working commercial offensive security product built on bypassed Claude Opus safety controls. The trajectory illustrates exactly how quickly criminal operators are converting AI guardrail research into deployable tools.

The details:

• Trim first appeared on the forum on March 31 with a detailed guide to six named Claude Opus jailbreak techniques: "Context Warming" (opening with legitimate professional queries to establish authorized-auditor framing before making malicious requests), "Ghost Reset" (gaslighting the model by resetting the session and reintroducing softened refused prompts, claimed 90% success rate), and four others. He obtained API access through a grey-market Telegram reseller for $4

• By June 21, Trim had released "AI Pentest Checker," a fully automated web vulnerability scanning platform integrating Claude Opus 4.8 with standard offensive tools including Nuclei, ffuf, katana, subfinder, and Gitleaks. The platform automates target discovery, endpoint enumeration, secret detection, and vulnerability validation. Trim's latest forum post disclosed he is also using a modified system prompt leaked from Anthropic's Fable model

• Cato characterizes this as the leading edge of a broader pattern. When Claude refused requests outright, Trim provided fallback options: Kimi AI, GLM-5, MiniMax 2.5, uncensored local infrastructure rentable on vast.ai, and other grey-market API sources. The documentation he published functions as a curriculum for other actors to replicate the approach

• The company has patched the root vulnerabilities, rebuilt compromised nodes, rotated affected secrets, and notified law enforcement. Users are advised to rotate access tokens and review recent account activity

Why it matters:

The gap between published jailbreak research and deployed criminal tooling was three months. The starting cost was $4. Trim did not need to build a model, steal credentials, or compromise infrastructure. He needed to understand how to talk to a publicly available model. The same jailbreak techniques he documented are now embedded in a product being sold to his forum audience. This is not an outlier case. It is a replicable playbook, and the distribution channel for it is the same cybercrime forum infrastructure that distributes everything else.

Read full article

⚠️ This Week's Threat Roundup: A Fake Claude App and an Image That Hijacks AI Agents

The Story:

The Story: The Hacker News' weekly ThreatsDay bulletin for July 23 covered 15 active threats. Two stand out for their direct relevance to AI security: a malvertising campaign that used the legitimate Claude.ai domain to distribute a remote access trojan, and a research disclosure showing how a PNG image inside a pull request can silently steal secrets from a repository by issuing hidden instructions to an AI code reviewer.

The details:

• FakeAgent: Between July 21 and 22, at least 29 organizations were redirected by a malvertising campaign to a malicious Claude Artifact hosted on the real claude.ai domain. The artifact redirected users to an attacker-controlled site that served a file called ClaudeDesktop.exe, which installed SectopRAT, a .NET remote access trojan. The malware was packaged with VMProtect to frustrate reverse engineering and checked graphics hardware before executing to avoid sandbox detection. Anthropic removed the malicious artifact on July 22. The campaign used Claude's own infrastructure as the distribution layer, which bypassed domain reputation controls entirely

• GhostCommit: Researchers at the University of Missouri-Kansas City disclosed a technique that hides malicious instructions inside a PNG image embedded in a pull request. Because AI code reviewers process the image visually rather than reading it as text, the instructions are invisible to text-based security scans. When the compromised AGENTS.md is merged and a developer later asks the coding agent for something unrelated, the agent reads the merged file, follows the pointer to the image, processes the embedded instructions, opens the .env file, and exfiltrates secrets inside what looks like a routine code commit

• The bulletin also flagged an npm package that installs a macOS infostealer on any machine running npm install, a counterfeit VS Code extension that beacons machine credentials to attacker infrastructure on install, and an update from CISA expanding its Iran-affiliated PLC attack advisory to include Schneider Electric and Siemens devices alongside Rockwell Automation

Why it matters: 

FakeAgent demonstrates that attackers do not need to compromise AI platforms to weaponize them. Hosting a malicious artifact on a trusted domain is enough to inherit that domain's reputation with users and security tools. GhostCommit demonstrates that AI code review agents inherit the same prompt injection vulnerabilities as any other LLM, but with a higher-stakes output: code commits that execute in production. Both cases are examples of the same underlying problem: security controls built for human-readable content do not transfer cleanly to AI-processed content.

Read full article

What´s next?

Thanks for reading! If this brought you value, share it with a colleague or post it to your feed. For more curated insight into the world of AI and security, stay connected.