In a cloud environment such as Kubernetes, attackers tend to target worker nodes with relatively powerful computing capabilities to gain benefits from hijacking computing resources (CPUs, GPUs, or FPGAs) on the nodes, as demonstrated by recent cases of crypto hijacking [1] [2]. If the target of the attack is an enterprise, the attack can affect the overall productivity and generate additional costs for the enterprise. Building a trusted container platform is crucial to mitigating such attacks.
In this
blog, we propose a trusted container framework with innovative
technologies to ensure the security of your container platform. With a five-level
model, the proposed framework can help address security concerns in all hardware
and software levels of a container-based platform.
We first introduce
the five levels of the security model by describing the building blocks of a
trusted container platform. Next, we elaborate on the technologies used, including
Intel® Security Libraries for Data Center (Intel® SecL – DC), Kubernetes
built-in policies, and security features proposed by QCT. Then, we present the
reference architecture diagram and introduce the role of each node in the
trusted container platform.
Building Blocks of the Trusted Container Platform
Figure 1. Building blocks of the trusted container platform
Figure 1 illustrates the building blocks of the trusted container platform with five color schemes, representing a five-level security model that can help ensure the platform stays in a trusted state with a chain of trust measurement measures. From the platform hardware all the way to orchestration software, the security model helps provide the following five levels of protective measures as follows:
- Ensuring the host is running with a valid firmware image to protect the system before secure boot starts.
- Ensuring the host uses trusted boot software only.
- Ensuring all the bootloaders are signed with a trusted key. Ensuring all sensitive data are encrypted and can only be decrypted by a trusted remote server or a Trusted Platform Module (TPM).
- Ensuring all the kernels and device drivers are signed with a trusted key.
- Ensuring the integrity of the host sensitive data. Ensuring that only authenticated and authorized users can launch the workload to a trusted host under specific network rules. Also, ensuring that the container image can only be pulled and decrypted by the nodes that meet specific trusted policies.
Technologies
The key concept of a trusted computing environment is to verify the integrity of the firmware and software components in the underlying platform. QCT partnered with Intel to define the five levels of the security model, spanning from the underlying hardware to the software level, in order to help ensure the security of the container platform at all times, from booting the physical server to monitoring platform scheduler and management. Table 1 lists the five security levels, the protected components, and the corresponding technologies applied.
Hardware Level Security
Intel® Boot Guard is used for boot integrity protection. To help ensure only authorized operating systems running on the system and to prevent threats such as malware, a rooted protected hardware infrastructure based on one-time programmable fuses (Field Programmable Fuses, FPFs) is adopted. These fuses are located in the Platform Controller Hub (PCH) and resist the execution of unauthorized Initial Boot Block (IBB). The computer will cease and refuse to boot after the validation fails.
UEFI Secure Boot is designed to add a layer of protection to the pre-boot process. After Secure Boot is activated, the firmware checks for the presence of a cryptographic signature on any EFI program it executes. If the signature is forbidden or does not match any key held in the platform’s NVRAM, the firmware will refuse the execution of the program. The signatures (of Boot Loader, Kernel images, and device drivers) are verified by UEFI Secure Boot during booting. They do not stop boot path manipulation, but only prevent the system from executing a modified boot path once the modification occurs.
Network Bound Disk Encryption is a subcategory of Policy-Based Decryption (PBD) that defines an automated system for the provisioning and acquisition of host decryption keys to the encrypted storage. That is, the NBDE allows binding encrypted volumes to a specific network server or TPM to boot without a password, which can prevent the disk from being decrypted, deleted, or stolen.
Software Level Security
Intel® Security Libraries for Data Center (Intel® SecL – DC)[3] is an open-source remote attestation implementation that consists of a set of building blocks that utilize Intel security features to discover, attest, and enable critical foundation security and confidential computing use-cases. It provides a consistent set of APIs for easy integration with cloud management software as well as security monitoring and enforcement tools.
To perform trust evaluations, once Intel® SecL – DC applies the remote attestation fundamentals, a platform data collection service and an efficient verification engine are maintained along with standard specifications of trust evaluations. These trust evaluations are then used to manage different trust and security policies applied to any workload, such as workload scheduling.
Open Security Content Automation Protocol (OpenSCAP) framework is a collection of specific standards that enables automated vulnerability management and configuration compliance. As part of Red Hat Enterprise Linux, OpenSCAP provides practical security hardening advice and remediations for Red Hat technologies, helping to meet security compliance requirements and thus, making it easier to obtain security certifications and accreditations. The adopted OpenSCAP Podman is used to evaluate the container and container image with a specific security baseline or compliance.
Auditd is the userspace component to the Linux Auditing System. It provides kernel-resident logging system calls and userspace tools to collect and view the logs. Auditd writes the logging records to disks and is configurable to allow control over what information can be written to the logs. System administrators use Auditd to define which actions on the server are to be monitored and to what extent. Auditd can track security-relevant events and detect any misuse or unauthorized activities by inspecting these log files. The administrators then take action to prevent these violations of system policies.
Kubernetes Container Platform is a production-grade container orchestration system. It provides a highly portable, extensive, and available architecture that hosts containerized applications. Kubernetes is widely used to automate deployment as well as to scale, maintain and validate the operation of applications. QCT adopts many of Kubernetes’ built-in security features, including Role-Based Access Control (RBAC), Network policies, and pod security policies to manage the resource accessibility and network traffic of the Kubernetes cluster.
Reference Architecture
Figure 2 illustrates the reference architecture of the trusted container platform used for testing and validation. The components are described in detail below.
Node Role
The architecture consists of five nodes, one Attestation Integrity Assurance Service (AIAS), one Kubernetes Worker node, and three Kubernetes Master nodes for high availability.
AIAS
The central AIAS acts as a root certificate authority and remote attestation authority, verifying the platform security technologies like Intel® Trusted Execution Technology (Intel® TXT), Intel® Boot Guard, and UEFI Secure Boot extend measurements of platform components (such as the system BIOS/UEFI, OS kernel, etc.) to a TPM as the server boots. AIAS is also responsible for host disk encryption and decryption. It discovers, attests, and enables system integrity by leveraging Intel® SecL – DC for confidential computing use cases.
Kubernetes Master
Kubernetes Master is responsible for the creation, scheduling, and management of all objects specific to the Kubernetes platform. The Kubernetes scheduler works with Intel® Security Libraries for Data Center with standard Custom Resource Definition (CRD) to make Kubernetes aware of Intel® SecL – DC security attributes and make them available for pod orchestration. In this way, a security-sensitive pod can be launched only on “Trusted” Kubernetes Worker nodes, or Kubernetes Worker nodes that match asset tag values; that is, the workload will be scheduled to a trusted node according to customized Kubernetes labels and the trusted state of attestation report from the AIAS server.
Kubernetes Worker
Kubernetes Worker node handles users’ containerized applications and data, which can be the main target for potential attackers. The trust agent resides in the Kubernetes worker node and enables both remote attestation and the extended chain of trust capabilities. The Agent maintains ownership of the server’s TPM, allowing secure attestation quotes to be sent to the AIAS. It serves to report on platform security capabilities and platform integrity measurements.
Conclusion
Building a trusted container platform is fundamental to mitigating advanced attacks to cloud and Kubernetes clusters. This blog presents a reference architecture for establishing a trusted container platform to support regulated or sensitive workloads, and introduces the technologies used to construct such prototype platforms. In the future, we will conduct more detailed functionality verification regarding security features, test methodology, and topology. Please stay tuned to our updates.