Tencent warns AI still fails in real world settings
Study finds weak context learning across leading AI systems.
New research from Tencent says leading AI systems in China and the US struggle in complex real world environments. Researchers argue weak context learning leaves models brittle outside controlled settings.
The study was co-authored by Tencent scientists and academics in China, led by former OpenAI researcher Vinces Yao Shunyu. Tests showed models often fail despite having all the necessary information.
Researchers created a new benchmark to assess how AI adapts to changing situations in China and the US. Results showed low average scores across 19 major models, indicating poor real world reliability.
Top performers included systems from Anthropic and OpenAI, though scores remained modest. Tencent said better context use could eventually come from deeper integration with WeChat in China and the US.
Would you like to learn more about AI, tech and digital diplomacy? If so, ask our Diplo chatbot!