IBM Cloud vulnerability allows for backdoor on bare metal servers
By Alfred Vergara on March 8, 2019
Researchers at Eclypsium on February 26, 2019 reported on a vulnerability in International Business Machines (IBM) Softlayer–now IBM Cloud–that allowed for a malicious user to plant a backdoor on servers. This backdoor would allow a malicious user to manage the server hardware after the use period has ended with the cloud service provider.
What is IBM Cloud?
IBM Cloud is a cloud service provider that allows for companies to purchase access to server capabilities over the Internet. Cloud service providers are attractive because clients do not need to manage hardware, and it is easily scalable if level of needs change. IBM cloud currently boasts usage by companies in over 20 industries, and had a 2.6% market share of a $40.8 billion cloud service industry in 2017.
Hypervisor Cloud Servers
These cloud servers may have multiple “tenants” on a physical server. These tenants are separated using a hypervisor, which is a device that allows for the virtualization of multiple machines on a single piece of hardware. Hypervisors allow a cloud service provider to fully utilize a physical service, and to easily move “tenants” in and out through reimaging processes.
Bare Metal Cloud Servers
These cloud servers have a single tenant on a physical server, which means that the whole mansion belongs to the client. Bare metal cloud server clients do not worry about other tenants taking up physical capabilities of the server (which can occur), and they are able to have access to the physical hardware of the server. This allows a client to have the benefits of a physical data center without building one on the premises; they do not assume responsibility of maintenance and storage of the physical infrastructure.
The Vulnerability
Since bare metal cloud service providers like IBM Cloud allow a client to have access to settings and configuration of the physical device a malicious user is able to access the Baseboard Management Controller (BMC). This device is a small chip that is present on server motherboards that allow an administrator to remotely access the server. This includes capabilities to access the file system, analyze log files, install new operating systems, update server firmware, and access outside networks. Access to the BMC is handled by the Intelligent Platform Management Interface (IPMI), which is a service that handles communication between the BMC and the server.
Researchers at Eclypsium acquired access to a piece of hardware from IBM Cloud, flipped a bit in the configuration file of the BMC, and added an administrator user to the IPMI. After relinquishing control and allowing IBM Cloud to go through their reacquisition process (presumably resetting the device), Eclypsium reacquired the same piece of hardware. (Verification of the hardware was done by finding the product serial numbers in both cases.) Eclypsium checked the hardware afterwards: the IPMI account was not present, but the bit in the configuration file of the BMC was still flipped. This indicated that IBM Cloud was not reflashing their BMCs. This could allow a user to upload malicious firmware to the BMC that would allow backdoor access to the server after their contract period. This allows a malicious user to “break-and-enter” into the server when another person is using it.
Impact
Bare metal cloud servers allow a client direct access to the server’s physical hardware. This is a service not limited to IBM Cloud, which was the target of the case study, but also by Amazon Web Services through their EC2 Bare-Metal services. Amazon Web Services is the largest market share holder of cloud services at 41.5% of the market. The lack of industry standards in securing cloud services may allow an attacker to disrupt applications, exfiltrate data, and deploy ransomware on cloud service clients.
Mitigation
IBM Cloud claimed that it had recognized the vulnerabilities and malpractice, and claimed that it will force all BMCs to be reflashed. This is a process that is different from reinstallation–it is a complete wipe and reinstallation of firmware. When Eclypsium checked on the same day, after reacquiring the same device for a third time, the bit was still flipped.
As a cloud service customer, it is important to follow a few guidelines:
- Evaluate service provider vulnerabilities before use
- Check for backdoors
- Reflash the firmware of new hosts
- Monitor firmware for changes
As a cloud service provider, it is important to consider the following:
- Check for modified firmware during reacquisition
- Reflash firmware after reacquisition
- Check for hardware tampering in the supply chain
Sources
https://www.pluralsight.com/blog/it-ops/what-is-hypervisor
https://www.businessnewsdaily.com/4982-cloud-vs-data-center.html
https://www.inetservices.com/bare-metal-cloud-servers/
https://www.welivesecurity.com/2017/10/19/malware-firmware-exploit-sense-security/
https://eclypsium.com/2019/01/26/the-missing-security-primer-for-bare-metal-cloud-services/
https://www.securityweek.com/hackers-can-plant-backdoors-bare-metal-cloud-servers-researchers