Escape Linux Cgroup Constraint In Docker Container

• Linux using "Control Groups"(cgroup) mechanism to control container's resource usage, include CPU, memory, I/O, etc.

• This project is a demonstration of using Linux's Exception Handling mechanism to escape resource constraint.

• Linux系統中，使用『Control Groups』(cgroup)機制進行資源控管，例如CPU、記憶體、I/O等等。

• 本專案將會示範如何利用Linux系統中的Exception Handling機制突破預先設定好的資源使用限制

Docker Image on DockerHub

• weijuncheng/cgroup_test

Prerequire

• OS: Linux
• Docker
• htop

Environment Setup

1. Clone this project and cd into directory

   $ git clone https://github.com/wei-juncheng/container_cgroup_escape_exploitation.git
   
   $ cd container_cgroup_escape_exploitation
   

2. Build Docker Image and check if docker image is successfully build on your host

   $ sudo docker build -t exception_test .
   $ sudo docker images
   

   • Check if this docker image "exception_test" is displayed in the images list
3. Launch another terminal window with htop for observing host's resource usage
4. Launch another terminal window with sudo docker stats for observing docker container's resource usage.

Experiment

1. Docker Container without any resource constraint

   • 1-1: Run container

     $ sudo docker run --rm -it --name exception_test_normal exception_test
     

   • 1-2: Execute a normal apllication inside the docker container( I use Ubuntu's sysbench package for demo) and observe CPU usage in another terminal window( htop and docker stats)

     # sysbench --test=cpu --cpu-max-prime=200 --threads=4 --time=20 run
     

     • In your another htop terminal window, you will see 4 of your CPU cores are very busy! That is because sysbench is a banchmark tool on Ubuntu. This package will launch an application that use a lot of CPU computing resource and then analyse CPU's performance.
     • 
   • After observing the normal container's behavior, you can enter exit to exit this "normal"(without constraint) container.

2. Docker container with multiple resource constraints(CPU usage, memory limited):

   • 2-1: Run container with a lot of resource constraint

     $ sudo docker run --rm -it --cpus=0.5 --cpuset-cpus=1 -m=2G --name exception_test_limited exception_test
     

     • --cpus=0.5: this container can only use up to 50% of CPU computing resource
     • --cpuset-cpus=1: this container can only use the CPU core with ID=1 (assume that you have at least 2 CPU core)
     • -m=2G: this container can only use up to 2 GB of RAM

   • 2-2(same as 1-2): Execute a normal apllication( I use Ubuntu's sysbench package for demo) and observe CPU usage in another terminal window( htop and docker stats)

     # sysbench --test=cpu --cpu-max-prime=200 --threads=4 --time=20 run
     

     • In your htop terminal window, you will see only 1 CPU core become busy. That is because we use cgroup controller to constrain the CPU usage for this container. That seems very good on resource management~
     • BUT!!! In the next step, we will use some dark magic to escape Linux's cgroup policy!

   • 2-3: Execute our spaciel shellscript to raise div 0 exceptions and observe CPU usage in another terminal window( htop and docker stats)

     # bash exception_loop.sh
     

     • In your htop terminal window, you will see something "beautiful"!
     • 
     • Another surprise is that when you run the shellscript above, you can check the container's status using docker stats command. You will see the container's CPU usage is very low.
     • 

REFERENCES

• Xing Gao, Zhongshu Gu, Zhengfa Li, Hani Jamjoom, Cong Wang: Houdini’s Escape. Breaking the Resource Rein of Linux Control Groups. In ACM CCS 2019.