The GSMA has identified significant limitations around the ability of current AI models to address telecom-specific queries
Ahead of Mobile World Congress Barcelona 2025, the GSMA Foundry, the association’s innovation hub, announced the launch of the GSMA Open-Telco LLM Benchmarks, which it described as an “open-source community aimed at improving the performance of large language models (LLMs) for telecom-specific applications” by providing “an industry-first framework for evaluating AI models in real-world telecom use cases.”
The GSMA has identified significant limitations around the ability of current AI models to address telecom-specific queries, noting that OpenAI’s fourth iterations of its GPT foundation model series GPT4 scored less than 75% on TeleQnA, a comprehensive dataset designed to assess LLM telecom knowledge, and less than 40% on 3GPPTdocs Classification, a dataset based on 3GPP standards documentation. Further, Microsoft’s Phi2 model scored only 10% on MATH500, a benchmark of 500 general math questions.
The GSMA Open-Telco LLM Benchmarks, which boasts initial support from Deutsche Telekom, LG Uplus, SK Telecom, Turkcell and Huawei, is expected to close these knowledge gaps in AI models by providing a standardized benchmarking framework and transparent, open evaluations of them across telecom capabilities, energy efficiency and safety. The resulting benchmarks will be hosted on Hugging Face’s open AI platform to “ensure transparency and encourage community engagement.”
“Today’s AI models struggle with telecom-specific queries, often producing inaccurate, misleading or impractical recommendations,” said Louis Powell, the head of AI initiatives at the GSMA. “By creating an industry-wide set of benchmarks, we’re not only improving model performance but also ensuring AI in telecoms is safe, reliable and aligned with real-world operational needs.”
SK Telecom’s involvement in the GSMA Open-Telco LLM Benchmarks is notable as the telco has been working with AI company Anthropic to fine-tune the latter’s Claude LLMs “to best meet the needs of telcos,” according to a press release from Anthropic. The AI company said further that it “will leverage SKT’s domain experience in telecommunications in order to make the model optimized for a wide variety of telco applications, including customer service, marketing, sales, and interactive customer applications.”
Of the latest news, SK Telecom’s Head of AI Tech Collaboration Office Eric Davis commented: The introduction of GSMA Open-Telco LLM Benchmarks marks a pivotal milestone for the telecommunications industry in its pursuit of tangible AI benefits. By establishing a standardi[z]ed evaluation framework, we’re simultaneously driving innovation and ensuring AI solutions deliver the robustness, reliability and precision that our rapidly evolving sector demands.”
Abu Dhabi’s Khalifa University and the Linux Foundation are additional GSMA Open-Telco LLM Benchmarks partners.
