February 10, 2025 issue
- Threat Actors Caught Abusing Large Language Models
- PowerSchool Breach Results in Large-scale Compromise of Student and Teacher Data
- Fear, Uncertainty, and Doubt: What to do about DeepSeek?
Threat Actors Caught Abusing Large Language Models
Threat actors are leveraging web-based large language models (LLMs) to enhance their productivity in research tasks across the various phases of the cyber kill chain. Providers of these models, such as OpenAI and Google, have been monitoring this behavior. They are tracking malicious usage by known threat groups, using this information to disrupt malicious campaigns, and gain insights into the operations of these threat actors.
On February 14th, 2024, OpenAI reported that they had disrupted the malicious use of AI by nation-state advanced persistent threats (APT). Six months later they disrupted a covert Iranian influence operation. And most recently on January 29th, 2025, Google's Threat Intelligence Group (GTIG) published an article revealing that APTs and influence operation actors are leveraging Google Gemini.
According to GTIG, APT groups from over 20 countries have utilized Gemini, with most of the data contributions coming from China and Iran. While there is overlap in the usage of Gemini among nation-state actors, specific groups have employed it in unique ways. For instance, North Korean APTs have been observed generating cover letters from job postings, likely to support their clandestine IT worker schemes.
GTIG findings align with that of others, and indicate that while AI is beneficial for attackers, it has not yet become the transformative hacking tool it is often depicted as. Currently, threat actors are mainly using Gemini to identify targets and develop tools and tradecraft, rather than employing the model for direct malicious activities. GTIG suggests that this usage likely enhances the productivity of advanced threats, increasing their frequency and volume, and also assists less advanced threat actors in their learning processes.
Currently, APTs appear to use popular LLMs the same way everyone else does, that is, to speed things up. However, as the AI landscape evolves, we can expect threat actor usage to evolve alongside it.
PowerSchool Breach Results in Large-scale Compromise of Student and Teacher Data
On December 28th 2024, PowerSchool, a leading cloud-based software provider for the education sector, discovered an incident involving unauthorized access to their PowerSource customer support portal. This breach led to the compromise of sensitive student and teacher data held in its Student Information System (SIS) database. In an extortion note, the unidentified threat actor claimed to have exfiltrated data from 6,505 school districts, affecting 62.4 million students and 9.5 million teachers across the US, Canada, and other countries.
The threat actor gained access using compromised credentials for a PowerSource customer support portal account that was not protected with multi-factor authentication (MFA). They then leveraged the "export data manager" tool, used by PowerSchool engineers to support customers, to export the SIS database tables containing student and teacher data. A custom script was used to rapidly iterate through this process and exfiltrate data from multiple school districts automatically. Upon discovering the breach, PowerSchool removed access to the affected portal and reset the credentials for all PowerSource customer support portal accounts.
PowerSchool confirmed that no encryption was deployed, but the threat actor did attempt to extort them by threatening to disclose the data, mirroring other encryption-less attacks which have been observed in recent years. In a bid to prevent dissemination of the data, a ransom was paid, and while PowerSchool received video showing what appeared to be the deletion of the data, they continue to monitor the dark web for potential leaks and have offered credit monitoring services to impacted adults and identity protection services for impacted minors.
Currently, 13 education organizations in Alberta have publicly disclosed being impacted by this incident. The stolen data includes highly sensitive information such as names, addresses, phone numbers, Social Insurance Numbers (SINs), grades, medical information, and passwords. A widely cited guide, provided by a PowerSchool customer, is available to help organizations identify any impact to their PowerSchool instances.
Fear, Uncertainty, and Doubt: What to do about DeepSeek?
On January 20th, 2025, the Chinese AI start-up DeepSeek launched its new R1 LLM and quickly rose in popularity. DeepSeek claimed that its model competes with the top Western AI models at a fraction of the cost. This claim led to a trillion-dollar sell-off of publicly traded AI stocks. The market reaction was driven by the belief that computing power is central to modern AI, and DeepSeek's ability to achieve competitive results with significantly less computing power challenged this notion.
In the weeks following its release, DeepSeek faced several security and privacy concerns. Privacy watchdogs began investigating DeepSeek’s data handling practices, and Texas banned DeepSeek along with other Chinese social media applications, citing security concerns. OpenAI accused DeepSeek of distilling their model to build its own. Additionally, on January 29th, the Cloud Security platform Wiz discovered an exposed DeepSeek database leaking sensitive information. The model also exhibited bias, refusing to answer questions that are contentious for the Chinese Communist Party (CCP) or citing the CCPs official stance.
Many of the concerns surrounding DeepSeek and its R1 model are based on fear, uncertainty, and doubt (FUD) and are exaggerated. The risk posed to the average organization is no different than that posed by ChatGPT or Gemini. None-the-less, the accusations, the investigations, and the banning of DeepSeek have left many wondering what they should do.
The answer to this question has been clouded by FUD, but it ultimately depends on your organization and what they deem acceptable use. The rules you apply to LLMs like ChatGPT should also apply to DeepSeek. If your organization is concerned about Chinese national intelligence laws and data storage in China, using DeepSeek locally to ensure that sensitive data remains within your organization's control, or opting not to use it at all, may be the best approach.