Lee Gyu-lee is a business writer at The Korea Times, focusing primarily on IT & telecommunications, the Ministry of Trade, Industry and Energy and KOTRA. Prior to this, she has covered a wide range of cultural news, from film, television and K-pop to lifestyle and fashion.
Motif challenges national AI model project after elimination
Second Vice Science Minister Ryu Je-myung speaks during a briefing at Government Complex Seoul, Aug. 18. Yonhap
By Lee Gyu-lee
AI startup questions evaluation criteria despite benchmark-leading model
Artificial intelligence (AI) startup Motif Technologies has filed an objection to its elimination from the second stage of evaluations for the government’s project to develop a national AI foundation model, calling for the disclosure of detailed scores and evaluation criteria and a reconsideration of the results.
The company said Thursday it had submitted a formal objection to the Ministry of Science and ICT, noting that its Motif 3 model posted the highest score among participating models on a global AI benchmark but received the lowest overall score in the government’s evaluation.
“Motif 3 received 47 points in the overall performance index, earning the highest score among the participating models, but was eliminated with the lowest score in the government’s results announcement,” the company said. “We ask for a detailed disclosure of the evaluation scores and for a reassessment.”
Motif said its objection was not intended to overturn the results or seek to advance to the third stage. Instead, it said it wanted greater transparency over how the project evaluates AI models, and hoped the process would clarify the project’s purpose and help find a direction that all competitors “can agree on and improve.”
The company’s Motif 3 model scored 47 on the Artificial Analysis Intelligence Index (AAII), compared with 37 for Upstage, 35 for SK Telecom and 31 for LG AI Research. Despite its higher benchmark score in the category, Motif was eliminated when the second-stage results were announced earlier this month, while Upstage, SK Telecom and LG AI Research advanced.
Participants try SK Telecom-led consortium's artificial intelligence (AI) model A.X K1 at a presentation event for the national AI foundation model project at Coex in Southern Seoul, Dec. 30, 2025. Courtesy of SK Telecom
The government’s second evaluation allocated 40 points to benchmark performance, 35 points to expert assessments and 25 points to user evaluations. The ministry disclosed the highest and average scores for each category but did not release individual team rankings or detailed scores, saying the project aims to foster and expand the AI ecosystem rather than simply rank companies.
“We have been cautious about the scope of information disclosed, taking into account potential direct and indirect harm to companies and their willingness to disclose their scores,” the ministry said.
Motif questioned why its strong benchmark performance did not result in a higher overall score.
“Despite the significant 16-point gap between the models’ AAII scores, the reasons and background behind the conflicting results between major global benchmarks and expert evaluations need to be explained in greater detail so that companies and the industry can understand them,” it said.
Motif also challenged the benchmark scoring formula, saying the 16-point gap between its AAII score and LG AI Research’s was narrowed to about four points under the 25-point scoring component.
The company also asked the ministry to disclose the criteria and rationale behind expert evaluations and whether user testing was conducted blind to limit the influence of brand awareness or preconceived views on the results.
The ministry defended the criteria, saying the project prioritizes both model performance and practical use, with greater emphasis on model independence after the first-stage evaluation.
It said the weight given to usability in expert evaluations was raised to 15 points from 10, while 25 points were allocated to real-user testing to reflect the project’s goals, adding that the criteria were agreed upon with participating companies and announced publicly.
ICT Minister Bae Kyung-hoon also acknowledged that the project still has work to do to develop AI foundation models ready for immediate use in industry and academia.
“We need to fundamentally consider whether an evaluation and competition system established a year and a half ago remains valid amid AI trends that are changing every three to six months,” Bae said Thursday during the ministry’s meeting on the future of science, technology and AI in central Seoul.
“We will consider ways to improve the government’s investment strategy and evaluation system to develop world-class models that anyone can recognize.”
The ministry said it would review Motif’s objection under procedures and issue a final decision within 15 days.
The dispute adds to concerns over fairness and transparency that have surrounded the government-led project since its launch.
In January, the elimination of Naver Cloud and NC AI from the first stage triggered criticism after the government changed its original rules and allowed an additional team to enter the competition through a separate selection process.
Motif was later selected through that additional process, joining the project after the initial five teams had been chosen.