Coudflare has revealed that its systems were compromised last year, leading to source code being accessed by threat actors.
The attack took place on November 23, 2023, and was perpetrated by a nation-state actor who used credentials stolen during a breach of Okta.
Cloudflare admitted that it “failed to rotate” its credentials that were stolen during the Okta breach. No customer data or systems were affected during the incident, which Cloudflare attributed to its zero trust environment limiting the threat actor’s ability to move laterally.
Crowdstrike has done a detailed analysis, following which Cloudflare provided details of the incident on February 1, 2024. During the Okta breach on October 18, 2023, the attackers stole one service token and three service account credentials belonging to Cloudflare.
These provided the following access to the cloud provider’s systems:
- A Moveworks service token granted remote access into the firm’s Atlassian system
- A service account used by the SaaS-based Smartsheet application, which gave administrative access to Cloudflare’s Atlassian Jira instance
- A Bitbucket service account allowing access to the source code management system
- An AWS environment that did not have access to the global network or contained customer or sensitive data
The threat actor with the stolen credentials successfully accessed Atlassian Jira and Confluence using the Moveworks service token.
The Smartsheet service account was then used to access the Atlassian suite. Once in these systems, the attackers searched for details about the configuration and management of Cloudflare’s global network, accessing various Jira tickets.
The threat actor also created an Atlassian user account via the Smartsheet credential. This was to ensure they’d have persistent access to the Atlassian environment should the Smartsheet account be removed.
These downloaded repositories were almost all related to system configuration and management at Cloudflare, such as how identity works and remote access. This is likely to find ways to mount a subsequent attack on the provider.
The company has treated the 76 downloaded source code repositories as exfiltrated by the attackers.
The threat actor was detected at 16.00 UTC on November 23, leading to Cloudflare’s security team deactivating the Smartsheet service account 35 minutes later. The user account created by the attacker was then discovered and deactivated 48 minutes later, at 17.23.
The Sliver tool was removed at 11.59 on November 24, and the last known threat actor activity was at 10.44 on the same day.
Cloudflare revealed that the threat actor attempted to access a “myriad” of other systems on its network, but its presence was limited to the Atlassian suite. This included an attempt to access a data center that Cloudflare had not yet put into production in São Paulo, Brazil.
Cloudflare said that this failure to move laterally was due to its zero trust architecture, which enforced access controls, firewall rules, and the use of hard security keys.
Cloudflare instituted a project dubbed “Code Red” to harden all controls in its environment and secure against future intrusion. This included analyzing the 76 stolen source code repositories to remediate embedded secrets, vulnerabilities, and other ways in which an attacker could use them to mount a subsequent attack.
To prevent the attackers from finding a new way back in, cloudflare undertook a “comprehensive” remediation effort, including:
- More than 5000 individual credentials were rotated
- All test and staging systems were physically rotated
- Forensic triages were performed on 4893 systems
- Every machine in its global network, including all Atlassian products, were reimagined and rebooted
