Cloning Running Servers with Docker and CRIU by Ross Boucher

Docker containers encapsulate everything you need to describe and run a process, but the lifecycle of a process remains the same: it starts, it runs for a while, and then it ends. This talk will demonstrate how to combine Docker with a tool called CRIU to “roll-back” running processes to an earlier state. CRIU, which stands for Checkpoint & Restore in User Space, creates a complete snapshot of the state of a process, including things like memory contents, file descriptors, and even open tcp connections. It can be used for suspending and resuming processes, or live migrating them from one machine to another. Our developer tool, Tonic, uses it to allow developers to change their code in the middle of a program without restarting from the beginning. We’ll show how we use the Docker Remote API to do this in production thousands of times a day.

1. Cloning Running Servers
Ross
Boucher
@boucher

3. $ node
>> 2 + 2
4
>>
Browser Server
2 + 2
4

4. x = 1
1
++x
2
x == 2
true
[1]
[2]
[3]
x = 1
1
++x
2
x == 2
true
[4]
[2]
[3]

6. (Checkpoint Restore In Userspace)
CRIU

7. CRIU
+

8. Application
Server
Evaluation
Controller
Public Internet Internal Network Isolated Network
Container PoolUser’s Browser
Primary Docker Eval Docker
2 + 2
4

9. message: "evaluate"
nodeVersion: "4.x.x"
sources: [{
type: "source",
text: "var x = 2 + 2",
checksum: "a8efd"
}]
url: "/users/boucher/repositories/12345/branches/master"
2 + 2 Application
Server

10. message: "get-evaluator"
nodeVersion: "4.x.x"
checksums: [ "a8efd" ]
url: "/users/boucher/repositories/12345/branches/master"
2 + 2
Evaluation
Controller
Application
Server

11. function get_evaluator(configuration) {
if (cached_containers[configuration]) {
return cached_containers[configuration]
}
if (checkpoint_exists(configuration)) {
return restored_container(configuration);
}
return pooled_container(configuration)
}
2 + 2
Evaluation
Controller

12. message: "found-evaluator"
IP: "172.0.1.201"
port: "7777"
Evaluation
Controller
Application
Server

13. message: "establish-connection"
message: "get-source"
index: 0
message: "source-at-index"
index: 0
source: "var x = 2+2;"
message: "output"
value: 4
Application
Server

14. message: "output"
index: 0
value: 4
2 + 2 Application
Server4

15. message: "checkpoint-evaluator"
checksum: "a8efd"
url: "/users/boucher/repositories/12345/branches/master"
2 + 2
Evaluation
Controller
Application
Server

16. await container.checkpoint({
LeaveRunning: true,
ImagesDirectory: “/checkpoints/<document_id>/<checksum>/”
})
if (await container.changes().length > 0) {
metadata.image = await container.commit({ pause: false })
}
await fs.writeFile(metadata,
“/checkpoints/<document_id>/<checksum>/metadata.txt” )
2 + 2Evaluation
Controller

17. # current cli
$ docker checkpoint --leave-running --image-dir= /checkpoint/path
<container_id>
$ docker commit --pause=false <container_id>
# new cli
$ docker checkpoint --exit=false <container_id> <checkpoint_id>
2 + 2Evaluation
Controller

18. message: "get-source"
...
message: "source-at-index"
...
message: "finished"
(process exited)
Application
Server

19. message: "predict-evaluator"
checksums: [ "a8efd" ]
url: "/users/boucher/repositories/12345/branches/master"
2 + 2
Evaluation
Controller
Application
Server

20. function predict_evaluator(configuration) {
var cpPath = “/checkpoints/<document_id>/<checksum>/” ;
var metadata = await fs.readFile(metadata, cpPath + “/metadata.txt”);
var container = await docker.restore({
ImagesDirectory: cpPath
...
});
cached_containers[configuration] = container;
}
2 + 2Evaluation
Controller

21. $ docker create tonic/worker
<container_id>
# current cli
$ docker restore --force —image-dir= /checkpoint/path <container_id>
# new cli
$ docker start --checkpoint <checkpoint_id> <container_id>
2 + 2Evaluation
Controller

22. message: "evaluate"
nodeVersion: "4.x.x"
sources: [
{ type: "source", text: "var x = 2 + 2", checksum: "a8fed"},
{ type: "source", text: "x++", checksum: "b9ccc" }
]
url: "/users/boucher/repositories/12345/branches/master"
Application
Server
2 + 2
4
x++

23. function get_evaluator(configuration) {
if (cached_containers[configuration]) {
return cached_containers[configuration]
}
if (checkpoint_exists(configuration)) {
return restored_container(configuration);
}
return pooled_container(configuration)
}
2 + 2
Evaluation
Controller

24. •Drop any capabilities you don’t need
•Set CPU, memory, and network constraints
•User Namespaces
•Network Isolation
•Seccomp
Security Concerns

25. •AUFS errors
•CRIU failures
•Race conditions
•Zombie processes
•Docker daemon restarts
•Filesystem management
Security Concerns

26. github.com/boucher/dockworker
DockWorker

29. (and other potential use cases)
Container Migration

30. !Pre-compiled Release (based on Docker 1.10)
!Checkpoint/Restore Pull Request
!Saied Kazemi’s Linux Plumber’s Talk
!CRIU Homepage
!DockerCon Doom Demo
!Tonic Blog on checkpoint/restore
!DockWorker on Github
!Using P.Haul with Docker
!DockerScript is a cool tool we use to manage our images
Further Reading

31. Thanks!
@boucher • [email protected]