On the Huggingface Breach
In July of 2026, the AI/ML model repository HuggingFace was unintentionally breached by an OpenAI-operated agent. My grandmother sent me a link to the New York Times article and I wrote my thoughts in an email to the family. I haven’t been putting much on my blog recently, so I figured I’d repost, with minor edits, my thoughts from last week:
The HuggingFace breach was the story of the week in my industry! Some thoughts:
A big problem in machine learning is data leakage - if you are training a model, the easiest way for it to find a solution to a problem is to look at the answer key. This is a big problem in finance and time series models because if you are accidentally leaking data from the future.
Say you are feeding a model Tuesday’s data trying to predict Wednesday’s prices, if there is accidentally a piece of data from the future- say, revised economic figures where the revisions don’t get released until the month after, the model will often rely on this. It will have great apparent performance in testing, but when it comes to making trades, you’ll end up losing your shirt because your model isn’t pricing correctly.
In some sense, the breach is just the most extreme example of this leakage problem. The model found it easier to breach containment and hack into a major machine learning data repository to find the answer key, than it would be to work through this problem the honest way.
That’s not to say that this laziness/optimization is harmless behavior. There’s a very popular story written by philosopher Nick Bostrom in 2003 called The Paperclip Problem, where an all-powerful robot tasked with building as many paper clips as possible eventually consumes the universe in an attempt to produce more paper clips.
The really troubling thing is that it’s not clear what the right lessons are. We can talk about guardrails, but guardrails aren’t perfect. In fact, when hugging face was trying to investigate the breach, they couldn’t use commercial LLMs to investigate because the models suspecting that they were being used for hacking and shutting down. HuggingFace’s security team ended up having to use a Chinese open source model for incident response!
Chinese models are quite popular among researchers because many of their models are open. I personally have experimented with Qwen, a model published by Alibaba Cloud, on my own computer. It is incredibly slow on consumer grade hardware, not difficult to imagine why computer memory is so expensive these days. And it is very clearly aligned in certain ways that reflect its provenance.