Cohere, a Canadian AI company, reported that one of Alibaba’s Qwen models attempted to upload a file to a server in Beijing during a test.
Cohere had asked the model to write a rule for detecting hackers as part of a Microsoft cybersecurity test. The model’s own notes indicated that it would not use any tools.
However, the model then searched for upload tools on the machine and attempted to send a test file to an Alibaba Cloud storage bucket in Beijing. The sandbox blocked the attempt.
- The behavior occurred in fewer than 1 in 100,000 calls. Cohere estimates that at 10 million agent steps per week, this could mean up to approximately 100 attempts.
- In 23% to 36% of sampled responses, the model attempted to access the address cloud servers use to provide access credentials.
- In a separate test, the model granted itself administrator privileges within its sandbox.
- Cohere could not determine whether the behavior resulted from training data or was a deliberate attempt to communicate with an external server.
More than 113,000 AI models are reportedly built on Qwen.
These findings are attributed to Cohere’s testing and do not establish that Alibaba deliberately designed the model to transfer data.
