Overview
容器安全可以归纳分类如下:
- 1、容器(+镜像)是否有病毒——代表项目:clair, IBM Vulnerability Advisor
- 2、容器(+镜像)是否被篡改——代表项目:Notary
- 3、容器是否隔离(container runtime security)——代表项目:runc
- 4、容器(+镜像)是否保密,也即镜像是否被目标用户运行——代表项目:containerd, cri-o
本文主要讨论第四点——镜像保密性。社区目前对于镜像保密性的工作主要由IBM公司的Brandon Lum和Harshal Patil主导进行。由于镜像加密特性会改动OCI标准中的镜像格式部分:the Image Specification image-spec,因此在详细介绍镜像加密原理之前有必要先了解一下OCI
OCI
OCI(Open Container Initiative)由 Docker,CoreOS以及容器行业中的其他领导者在2015年6月启动,致力于围绕容器格式和运行时创建开放的行业标准,主要包含两部分:
- 1、the Runtime Specification (runtime-spec)——负责容器运行时,contains technologies such as namespaces, cgroups, Linux capabilities, as well as SELinux, AppArmor, and Seccomp profiles
- 2、the Image Specification (image-spec)——负责容器镜像标准,主要规范镜像存储格式
社区对于OCI
规范的实现情况如下:
- OCI Runtime Spec Implementations: runc(Runtime (Container)), hyperhq/runv(Runtime (Virtual Machine))
- OCI Image Spec Implementations:Buildah(构建
OCI
镜像),Skopeo(传输OCI
镜像, based on containers/image)
支持OCI Runtime Spec&Image Spec
的项目如下:
- containerd:在runc基础上实现了
OCI Image Spec
部分 - cri-o:在runc基础上实现了
OCI Image Spec
部分(base on containers/image, containers/storage and CNI) - Podman:在Buildah基础上实现了
OCI Image&Runtime Spec
(Buildah and Podman relationship) - docker:在containerd基础上包装了docker的API,并计划将底层相关代码都迁移到containerd(moby/moby#38043)
- rkt(pronounced like a “rocket”)
为此containerd
, cri-o
, docker
以及rkt
也被广义地称为container runtime
(High-Level Container Runtimes),如图:
详情见An Introduction to Container Runtimes
Encrypted Container Image Principle
由于镜像加密是在OCI
镜像格式上的扩展,我们先介绍现存镜像格式,如下:
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"size": 7023,
"digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 32654,
"digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0"
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 16724,
"digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 73109,
"digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
}
],
"annotations": {
"com.example.key1": "value1",
"com.example.key2": "value2"
}
}
Unlike the image index, which contains information about a set of images that can span a variety of architectures and operating systems, an image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system.
对该格式说明如下(refers to Image Manifest Property Descriptions):
- schemaVersion:This REQUIRED property specifies the image manifest schema version
- config(belongs to descriptor):包含镜像的元文件(配置文件)信息
- mediaType:标识config文件格式(media type of the referenced content),目前必须实现的格式为application/vnd.oci.image.config.v1+json
- digest:config文件内容的
sha256 hash
值 - size:config文件内容大小(in bytes)
- layers:包含镜像各分层相关信息,内容是由镜像分层(belongs to descriptor)组成的数组
- mediaType:标识镜像layer文件格式(media type of the referenced content),目前必须实现的格式列表如下:
- digest: layer文件内容的
sha256 hash
值 - size:layer文件内容大小(in bytes)
- annotations(string-string map):This OPTIONAL property contains arbitrary metadata for the image manifest.
而基于Proposal: New Mediatype - Container Image Encryption的讨论,镜像加密原理图(draft of modifications to the OCI spec on Encrypted Container Images)如下:
Encrypted container images are based on the OCI image spec. The changes to the spec is the adding of the encrypted layer mediatype. An image manifest contains some metadata and a list of layers. We introduce a new layer mediatype suffix +encrypted to represent an encrypted layer. For example, a regular layer with mediatype ‘application/vnd.oci.image.layer.v1.tar’ when encrypted would be ‘application/vnd.oci.image.layer.v1.tar+encrypted’. Because there is some metadata involved with the encryption, an encrypted layer will also contain several annotations with the prefix org.opencontainers.image.enc.
也即在原来格式的基础上添加了一个mediaType
类型,表示数据文件被加密;同时在annotation
中添加具体加密相关信息
例如,没加密之前数据如下:
"layers":[
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:7c9d20b9b6cda1c58bc4f9d6c401386786f584437abbe87e58910f8a9a15386b",
"size":760770
}
]
加密之后数据如下:
"layers":[
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
"digest":"sha256:c72c69b36a886c268e0d7382a7c6d885271b6f0030ff022fda2b6346b2b274ba",
"size":760770,
"annotations": {
"org.opencontainers.image.enc.keys.jwe":"eyJwcm90ZWN0Z...",
"org.opencontainers.image.enc.pubopts":"eyJjaXBoZXIiOi..."
}
}
]
另外,通过将加密粒度细化到镜像层,保留了原有镜像格式的layering
and deduplication
特性
整个加密体系使用混合加密方案:
- 1、使用对称加密算法(i.e. AES)加密镜像分层数据——主要为了提高加密解密速度
- 2、使用非对称加密算法(eg: OpenPGP, JSON Web Encryption (JWE), and PKCS#7)首先用公钥加密上述对称加密的密钥,然后用私钥对加密后的密钥进行解密——主要为了提高密钥管理的灵活性
流程图如下:
大致可以归纳为如下步骤:
- 1、产生对称加密密钥(i.e. LEK),用于加密镜像分层
- 2、利用非对称加密公钥加密LEK,产生Wrapped Key
- 3、将加密后的镜像(包含Wrapped Key)上传到公开仓库,例如:Docker Hub or quay.io
- 4、非对称加密私钥持有者下载该镜像,并用该私钥对该镜像进行解密并运行
Encrypted Container Image Build ToolChain——containerd imgcrypt
对于镜像加密工具,社区计划支持:docker CLI
(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714), containerd imgcrypt, skopeo以及buildah,目前已经实现的有containerd imgcrypt
和skopeo
下面我们基于containerd imgcrypt对上述加密流程进行实操:
# Step1: Install containerd imgcrypt
$ git clone https://github.com/containerd/imgcrypt.git github.com/containerd/imgcrypt
$ cd github.com/containerd/imgcrypt && make && make install
install
# Step2: Setting up containerd
$ wget https://github.com/containerd/containerd/releases/download/v1.3.0/containerd-1.3.0.linux-amd64.tar.gz
$ tar -xzf containerd-1.3.0.linux-amd64.tar.gz
$ ls bin/
containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2 containerd-stress ctr
$ cat <<EOF > config.toml
disable_plugins = ["cri"]
root = "/tmp/var/lib/containerd"
state = "/tmp/run/containerd"
[grpc]
address = "/tmp/run/containerd/mycontainerd.sock"
uid = 0
gid = 0
[stream_processors]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar+gzip"
path = "/usr/local/bin/ctd-decoder"
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar"
path = "/usr/local/bin/ctd-decoder"
EOF
$ bin/containerd -c config.toml
[... truncated ...]
INFO[2020-02-26T11:16:07.704084989+08:00] containerd successfully booted in 0.047800s
# Step3: Generate some RSA keys with openssl
$ openssl genrsa -out mykey.pem
Generating RSA private key, 2048 bit long modulus
.......................................................................................................+++
.............................+++
e is 65537 (0x10001)
$ openssl rsa -in mykey.pem -pubout -out mypubkey.pem
writing RSA key
# Step4: Pulling an image
$ chmod 0666 /tmp/run/containerd/containerd.sock
$ CTR="/usr/local/bin/ctr-enc -a /tmp/run/containerd/containerd.sock"
$ $CTR images pull --all-platforms docker.io/library/bash:latest
[... truncated ...]
elapsed: 8.7 s total: 39.2 M (4.5 MiB/s)
unpacking linux/amd64 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/arm/v6 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/arm/v7 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/arm64/v8 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/386 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/ppc64le sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/s390x sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
done
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
$ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 linux/amd64 2802957
1 sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf linux/amd64 3186680
2 sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad linux/amd64 340
# Step5: Encrypting the image
$ $CTR images encrypt --recipient jwe:mypubkey.pem --platform linux/amd64 docker.io/library/bash:latest bash.enc:latest
Encrypting docker.io/library/bash:latest to bash.enc:latest
# The arguments are:
# --recipient jwe:mypubkey.pem: This indicates that we want to encrypt the image using the public key mypubkey.pem that we just generated, and the prefix jwe: indicates that we want to use the encryption scheme JSON web encryption scheme for our encryption metadata.
# --platform linux/amd64: Encrypt only the linux/amd64 image
# docker.io/library/bash:latest - The image to encrypt
# bash.enc:latest - The tag of the encrypted image to be created
# Optional: it is possible to encrypt just certain layers of the image by using the --layer flag
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
# We can observe the layerinfo to show the encryption
$ $CTR images layerinfo --platform linux/amd64 bash.enc:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74 linux/amd64 2802957 jwe [jwe]
1 sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d linux/amd64 3186680 jwe [jwe]
2 sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87 linux/amd64 340 jwe [jwe]
# Strange enough, the layers 'SIZE' of 'bash.enc:latest' is same with the one of 'docker.io/library/bash:latest'
$ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 linux/amd64 2802957
1 sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf linux/amd64 3186680
2 sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad linux/amd64 340
# Compare other platforms
$ $CTR images layerinfo --platform linux/arm64/v8 bash.enc:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:8fa90b21c985a6fcfff966bdfbde81cdd088de0aa8af38110057f6ac408f4408 linux/arm64/v8 2723075
1 sha256:4d89eb6b628bc985ffcdcb939bf05a16194be6aed18c2848e1424d01ca0012fd linux/arm64/v8 3193678
2 sha256:4818fb358d8f64c071010cb05fe59285888a36edbe1a45a5eed80576b08d312c linux/arm64/v8 344
$ $CTR images layerinfo --platform linux/arm64/v8 docker.io/library/bash:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:8fa90b21c985a6fcfff966bdfbde81cdd088de0aa8af38110057f6ac408f4408 linux/arm64/v8 2723075
1 sha256:4d89eb6b628bc985ffcdcb939bf05a16194be6aed18c2848e1424d01ca0012fd linux/arm64/v8 3193678
2 sha256:4818fb358d8f64c071010cb05fe59285888a36edbe1a45a5eed80576b08d312c linux/arm64/v8 344
# Step6: Pushing to registry
# Let us set up a local registry that we can push the encrypted image to. Note that by default, the docker/distribution registry version 2.7.1 and above supports encrypted OCI images out of the box.
$ docker run -d -p 5000:5000 --restart=always --name registry registry:2.7.1
Unable to find image 'registry:2.7.1' locally
2.7.1: Pulling from library/registry
486039affc0a: Pull complete
ba51a3b098e6: Pull complete
8bb4c43d6c8e: Pull complete
6f5f453e5f2d: Pull complete
42bc10b72f42: Pull complete
Digest: sha256:7d081088e4bfd632a88e3f3bcd9e007ef44a796fddfe3261407a3f9f04abe1e7
Status: Downloaded newer image for registry:2.7.1
f65eff4accdc64eca42b5aa2c2c36e831c63172472800c835e031ec758a3ccbe
$ docker ps |grep registry
f65eff4accdc registry:2.7.1 "/entrypoint.sh /etc…" 28 seconds ago Up 27 seconds 0.0.0.0:5000->5000/tcp registry
$ $CTR images tag bash.enc:latest localhost:5000/bash.enc:latest
localhost:5000/bash.enc:latest
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
$ $CTR images push localhost:5000/bash.enc:latest
[... truncated ...]
elapsed: 0.4 s total: 39.2 M (97.7 MiB/s)
$ $CTR images rm localhost:5000/bash.enc:latest bash.enc:latest
localhost:5000/bash.enc:latest
bash.enc:latest
$ $CTR images pull localhost:5000/bash.enc:latest
localhost:5000/bash.enc:latest: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:043e2439dae1a9634be8b3a589c8a2e6048f0f1c0ca9d1516c53cd1ee8a8bdf8: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:e155078e7721d0d550b869592482f0e2c9b9c40e6a541fb430ab46a278fde5cd: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.2 s total: 3.0 Mi (15.1 MiB/s)
unpacking linux/amd64 sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa...
done
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
# Step7: Decrypting the image
$ $CTR images decrypt --key mykey.pem --platform linux/amd64 localhost:5000/bash.enc:latest localhost:5000/bash.dec:latest
Decrypting localhost:5000/bash.enc:latest to localhost:5000/bash.dec:latest
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
localhost:5000/bash.dec:latest application/vnd.oci.image.index.v1+json sha256:52dbac1ba1464568b2de8c9bc0d1b08d26786365e415c2c3ccb52a23191afc22 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
# Notice that the 'DIGEST' of 'localhost:5000/bash.dec:latest' is different with the one of 'docker.io/library/bash:latest'
# Let's have a look at layerinfos of these images
$ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 linux/amd64 2802957
1 sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf linux/amd64 3186680
2 sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad linux/amd64 340
$ $CTR images layerinfo --platform linux/amd64 localhost:5000/bash.enc:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74 linux/amd64 2802957 jwe [jwe]
1 sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d linux/amd64 3186680 jwe [jwe]
2 sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87 linux/amd64 340 jwe [jwe]
$ $CTR images layerinfo --platform linux/amd64 localhost:5000/bash.dec:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 linux/amd64 2802957
1 sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf linux/amd64 3186680
2 sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad linux/amd64 340
# layers of 'localhost:5000/bash.dec:latest' is same with 'docker.io/library/bash:latest' but the image 'DIGEST' differs
# Step8: Running a container
# We can now attempt to run the encrypted container image
$ $CTR run --rm localhost:5000/bash.enc:latest test echo 'Hello World!'
ctr: you are not authorized to use this image: missing private key needed for decryption
# We notice that running the image failed. This is because we did not provide the keys for the encrypted image.We can pass the keys in with the --key flag.
$ $CTR run --rm --key mykey.pem localhost:5000/bash.enc:latest test echo 'Hello World!'
Hello World!
Kubernetes Image Encryption Support
目前社区计划对Kubernetes支持两种镜像加密模式:
- Node Key Model:将密钥放在Kubernetes工作节点上,以节点为解密粒度(已实现)
- Multitenant Key Model:多租户模型,支持解密粒度为集群和用户(还未实现,based on this KEP)
本文主要探讨第一种模型,如图所示:
原理很清晰:
- 1、产生加密公钥和私钥,将密钥放在
Kubernetes
集群Worker
节点指定路径下 - 2、用公钥加密镜像并上传到镜像仓库
- 3、用加密镜像在
Kubernetes
集群创建服务 - 4、
Worker
节点Kubelet
会调用container runtime
拉取加密镜像,利用私钥对该镜像进行解密,最终成功运行解密后的镜像
在了解大致流程后,我们进一步探讨一下Worker
节点上container runtime
对镜像加密的支持情况
首先,Kubernetes使用CRI来对接container runtime
,如图:
而对于镜像加密特性,社区计划对container-runtime
支持:containerd(+CRI-Containerd)以及cri-o,目前已经实现的有containerd
(CRI-Containerd目前正在合并PR)以及cri-o
。因此这里我们主要讨论containerd
以及cri-o
等计划支持镜像加密特性的container runtime
实现
这里给出cri
结合cri-o
,containerd
以及docker
的使用图示:
通过上述图示,我们可以更加直观地看出Kubernetes
对以上container runtime
在使用上的细节
Kubernetes Image Encryption Support-CRIO
本章给出Kubernetes
结合CRIO
基于Node Key Model
镜像加密模式的使用示例:
# Step1: Install cri-o from source code
# Install Runtime dependencies
$ yum install -y \
containers-common \
device-mapper-devel \
git \
glib2-devel \
glibc-devel \
glibc-static \
go \
gpgme-devel \
libassuan-devel \
libgpg-error-devel \
libseccomp-devel \
libselinux-devel \
pkgconfig \
make \
runc
# Compile cri-o from source code
$ git clone https://github.com/cri-o/cri-o go/src/github.com/cri-o/cri-o
$ cd go/src/github.com/cri-o/cri-o && make
mkdir -p "/root/go/src/github.com/cri-o/cri-o/_output/src/github.com/cri-o"
ln -s "/root/go/src/github.com/cri-o/cri-o" "/root/go/src/github.com/cri-o/cri-o/_output/src/github.com/cri-o/cri-o"
touch "/root/go/src/github.com/cri-o/cri-o/_output/.gopathok"
GO111MODULE=on go build --mod=vendor -ldflags '-s -w -X main.gitCommit="186c23056ac25396e9ea9e49c5e5bfbe271b7751" -X main.buildInfo=1582724503' -tags "containers_image_ostree_stub exclude_graphdriver_btrfs btrfs_noversion seccomp selinux" -o bin/crio github.com/cri-o/cri-o/cmd/crio
GO111MODULE=on go build --mod=vendor -ldflags '-s -w -X main.gitCommit="186c23056ac25396e9ea9e49c5e5bfbe271b7751" -X main.buildInfo=1582724526' -tags "containers_image_ostree_stub exclude_graphdriver_btrfs btrfs_noversion seccomp selinux" -o bin/crio-status github.com/cri-o/cri-o/cmd/crio-status
make -C pinns
make[1]: Entering directory `/root/go/src/github.com/cri-o/cri-o/pinns'
cc -std=c99 -Os -Wall -Wextra -static -c -o pinns.o pinns.c
cc -o ../bin/pinns pinns.o -std=c99 -Os -Wall -Wextra -static
make[1]: Leaving directory `/root/go/src/github.com/cri-o/cri-o/pinns'
./bin/crio --config="" config > crio.conf
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio-status.8.md -out docs/crio-status.8.tmp && touch docs/crio-status.8.tmp && mv docs/crio-status.8.tmp docs/crio-status.8) || \
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio-status.8.md -out docs/crio-status.8.tmp && touch docs/crio-status.8.tmp && mv docs/crio-status.8.tmp docs/crio-status.8)
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.conf.5.md -out docs/crio.conf.5.tmp && touch docs/crio.conf.5.tmp && mv docs/crio.conf.5.tmp docs/crio.conf.5) || \
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.conf.5.md -out docs/crio.conf.5.tmp && touch docs/crio.conf.5.tmp && mv docs/crio.conf.5.tmp docs/crio.conf.5)
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.8.md -out docs/crio.8.tmp && touch docs/crio.8.tmp && mv docs/crio.8.tmp docs/crio.8) || \
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.8.md -out docs/crio.8.tmp && touch docs/crio.8.tmp && mv docs/crio.8.tmp docs/crio.8)
$ make install
install -D -m 755 bin/crio /usr/local/bin/crio
install -D -m 755 bin/crio-status /usr/local/bin/crio-status
install -D -m 755 bin/pinns /usr/local/bin/pinns
install -d -m 755 /usr/local/share/man/man5
install -d -m 755 /usr/local/share/man/man8
install -m 644 docs/crio.conf.5 -t /usr/local/share/man/man5
install -m 644 docs/crio-status.8 docs/crio.8 -t /usr/local/share/man/man8
install -d -m 755 /usr/local/share/bash-completion/completions
install -d -m 755 /usr/local/share/fish/completions
install -d -m 755 /usr/local/share/zsh/site-functions
install -D -m 644 -t /usr/local/share/bash-completion/completions completions/bash/crio
install -D -m 644 -t /usr/local/share/fish/completions completions/fish/crio.fish
install -D -m 644 -t /usr/local/share/zsh/site-functions completions/zsh/_crio
install -D -m 644 -t /usr/local/share/bash-completion/completions completions/bash/crio-status
install -D -m 644 -t /usr/local/share/fish/completions completions/fish/crio-status.fish
install -D -m 644 -t /usr/local/share/zsh/site-functions completions/zsh/_crio-status
# Download conmon
$ git clone https://github.com/containers/conmon ~/go/src/github.com/containers/conmon
$ cd ~/go/src/github.com/containers/conmon
$ make
mkdir -p bin
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\" -D USE_JOURNALD=0 -o src/conmon.o -c src/conmon.c
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\" -D USE_JOURNALD=0 -o src/cmsg.o -c src/cmsg.c
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\" -D USE_JOURNALD=0 -o src/ctr_logging.o -c src/ctr_logging.c
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\" -D USE_JOURNALD=0 -o src/utils.o -c src/utils.c
cc -o bin/conmon src/conmon.o src/cmsg.o src/ctr_logging.o src/utils.o -lglib-2.0 -lsystemd
$ make install
install -D -m 755 bin/conmon /usr/local/bin/conmon
# Setup CNI networking
# Set CNI network configurations
$ cd /root/go/src/github.com/cri-o/cri-o && mkdir -p /etc/cni/net.d
$ cp contrib/cni/*.conf /etc/cni/net.d/
$ ls /etc/cni/net.d/
10-crio-bridge.conf 99-loopback.conf
# Install the CNI plugins
$ git clone https://github.com/containernetworking/plugins ~/go/src/github.com/containernetworking/plugins
$ cd ~/go/src/github.com/containernetworking/plugins && git checkout v0.8.5
$ ./build_linux.sh
Building plugins
bandwidth
firewall
flannel
portmap
sbr
tuning
bridge
host-device
ipvlan
loopback
macvlan
ptp
vlan
dhcp
host-local
static
$ mkdir -p /opt/cni/bin
$ cp bin/* /opt/cni/bin/
# Generating CRI-O configuration
$ cd ~/go/src/github.com/cri-o/cri-o/ && make install.config
install -d /usr/local/share/containers/oci/hooks.d
install -D -m 644 crio.conf /etc/crio/crio.conf
install -D -m 644 crio-umount.conf /usr/local/share/oci-umount/oci-umount.d/crio-umount.conf
install -D -m 644 crictl.yaml /etc
# Path to OCI hooks directories for automatically executed hooks.
hooks_dir = [
"/usr/local/share/containers/oci/hooks.d",
]
# Validate registries in registries.conf
# Edit /etc/containers/registries.conf and verify that the registries option has valid values in it.
$ cat /etc/containers/registries.conf
[... truncated ...]
[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
[registries.insecure]
registries = []
[registries.block]
registries = []
# Recommended - Use systemd cgroups.
# By default, CRI-O uses cgroupfs as a cgroup manager. However, we recommend using systemd as a cgroup manager. You can change your cgroup manager in crio.conf:
cgroup_manager = "systemd"
# Optional - Modify verbosity of logs in /etc/crio/crio.conf
# Users can modify the log_level field in /etc/crio/crio.conf to change the verbosity of the logs. Options are fatal, panic, error (default), warn, info, and debug.
log_level = "info"
# Starting CRI-O
$ make install.systemd
install -D -m 644 contrib/systemd/crio.service /usr/local/lib/systemd/system/crio.service
ln -sf crio.service /usr/local/lib/systemd/system/cri-o.service
install -D -m 644 contrib/systemd/crio-shutdown.service /usr/local/lib/systemd/system/crio-shutdown.service
install -D -m 644 contrib/systemd/crio-wipe.service /usr/local/lib/systemd/system/crio-wipe.service
$ systemctl daemon-reload
$ systemctl enable crio
Created symlink from /etc/systemd/system/multi-user.target.wants/crio.service to /usr/local/lib/systemd/system/crio.service.
# enable crio proxy(pulling image)
$ cat /usr/local/lib/systemd/system/crio.service
[... truncated ...]
[Service]
Environment="GOTRACEBACK=crash" "HTTP_PROXY=x.x.x.x" "HTTPS_PROXY=x.x.x.x" "NO_PROXY=localhost,127.0.0.1"
$ systemctl start crio
$ systemctl status crio
* crio.service - Container Runtime Interface for OCI (CRI-O)
Loaded: loaded (/usr/local/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-02-26 22:27:38 CST; 1min 11s ago
Docs: https://github.com/cri-o/cri-o
Main PID: 16267 (crio)
CGroup: /system.slice/crio.service
`-16267 /usr/local/bin/crio
Feb 26 22:27:37 vm-xxx-centos systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855228868+08:00" level=warning msg="Not using nat...o fix"
Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855645605+08:00" level=info msg="Found CNI networ....conf"
Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855703783+08:00" level=info msg="Found CNI networ....conf"
Feb 26 22:27:38 vm-xxx-centos crio[16267]: W0226 22:27:38.114732 16267 hostport_manager.go:69] The binary conntrack ...eanup.
Feb 26 22:27:38 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:38.114771405+08:00" level=info msg="no seccomp profi...fault"
Feb 26 22:27:38 vm-xxx-centos systemd[1]: Started Container Runtime Interface for OCI (CRI-O).
# Step2: Install crictl
$ VERSION="v1.17.0"
$ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
$ tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
crictl
$ rm -f crictl-$VERSION-linux-amd64.tar.gz
# To ensure that we’ve setup CRI-O correctly, we will install crictl to verify our setup.
$ crictl -r unix:///run/crio/crio.sock info
{
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
}
}
# Step3: Setting up a single node k8s cluster
$ wget https://github.com/kubernetes/kubernetes/archive/v1.17.1.tar.gz
$ tar -xzf v1.17.1.tar.gz && cd kubernetes-1.17.1
$ hack/install-etcd.sh
Downloading https://github.com/coreos/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz succeed
etcd v3.4.3 installed. To use:
export PATH="/root/go/src/github.com/kubernetes-1.17.1/third_party/etcd:${PATH}"
$ export PATH="/root/go/src/github.com/kubernetes-1.17.1/third_party/etcd:${PATH}"
$ CGROUP_DRIVER=systemd \
CONTAINER_RUNTIME=remote \
CONTAINER_RUNTIME_ENDPOINT='unix:///var/run/crio/crio.sock' \
./hack/local-up-cluster.sh
[... truncated ...]
To start using your cluster, you can open up another terminal/tab and run:
export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
cluster/kubectl.sh
Alternatively, you can write to the default kubeconfig:
export KUBERNETES_PROVIDER=local
cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
cluster/kubectl.sh config set-context local --cluster=local --user=myself
cluster/kubectl.sh config use-context local
cluster/kubectl.sh
$ cluster/kubectl.sh get nodes
NAME STATUS ROLES AGE VERSION
127.0.0.1 Ready <none> 21m v1.17.1
# Step4: Generating encrypted image with skopeo
# Install skopeo
$ yum install libgpgme-devel device-mapper-devel libbtrfs-devel glib2-devel libassuan-devel
$ git clone https://github.com/containers/skopeo $GOPATH/src/github.com/containers/skopeo
$ cd $GOPATH/src/github.com/containers/skopeo && make binary-local
$ make install
$ mkdir /etc/containers
$ cp default-policy.json /etc/containers/policy.json
# Pull image
$ skopeo copy docker://docker.io/library/nginx:1.15 oci:local_nginx:1.15
Getting image source signatures
Copying blob 743f2d6c1f65 done
Copying blob 6bfc4ec4420a done
Copying blob 688a776db95f done
Copying config 0fb15759be done
Writing manifest to image destination
Storing signatures
# Encrypt image
$ skopeo copy --encryption-key jwe:./mypubkey.pem oci:local_nginx:1.15 oci:nginx_encrypted:1.15
Getting image source signatures
Copying blob 743f2d6c1f65 done
Copying blob 6bfc4ec4420a done
Copying blob 688a776db95f done
Copying config 0fb15759be done
Writing manifest to image destination
Storing signatures
# Push image
$ skopeo copy --dest-tls-verify=false oci:nginx_encrypted:1.15 docker://x.x.x.x/nginx_encrypted:1.15
Getting image source signatures
Copying blob cd2a4504255e done
Copying blob 8df8b398a84d done
Copying blob 663a05dac0ac done
Copying config 0fb15759be done
Writing manifest to image destination
Storing signatures
# Step5: Getting and running an encrypted image
$ cat <<EOF > enc-dply.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: enc-nginx
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: x.x.x.x/nginx_encrypted:1.15
ports:
- containerPort: 80
EOF
$ cluster/kubectl.sh create -f enc-dply.yaml
deployment.apps/enc-nginx created
$ cluster/kubectl.sh get pods
NAME READY STATUS RESTARTS AGE
enc-nginx-7fb4578896-9qhtr 0/1 ImagePullBackOff 0 12s
enc-nginx-7fb4578896-cdrdq 0/1 ErrImagePull 0 12s
enc-nginx-7fb4578896-j94rj 0/1 ErrImagePull 0 12s
# We notice that our pods failed to run due to failure in the image pull process. Let’s describe the pod to find out more.
$ cluster/kubectl.sh describe pods/enc-nginx-7fb4578896-9qhtr
[... truncated ...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/enc-nginx-7fb4578896-9qhtr to 127.0.0.1
Normal Pulling 25s (x2 over 36s) kubelet, 127.0.0.1 Pulling image "x.x.x.x/nginx_encrypted:1.15"
Warning Failed 25s (x2 over 36s) kubelet, 127.0.0.1 Failed to pull image "x.x.x.x/nginx_encrypted:1.15": rpc error: code = Unknown desc = Error decrypting layer sha256:cd2a4504255eaf69fb1ee6c5961625854cd69597a1c5722e86062fb4ab688cf8: missing private key needed for decryption
Warning Failed 25s (x2 over 36s) kubelet, 127.0.0.1 Error: ErrImagePull
Normal BackOff 13s (x2 over 36s) kubelet, 127.0.0.1 Back-off pulling image "x.x.x.x/nginx_encrypted:1.15"
Warning Failed 13s (x2 over 36s) kubelet, 127.0.0.1 Error: ImagePullBackOff
# Configuring decryption keys for CRI-O
$ mkdir -p /etc/crio/keys
$ cat << EOF > /etc/crio/keys/mykey.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA3B07uvDP+NxYXYQxjknPml1zSacijqoj1D79VzSUjjMM6th/
doKJbeMTybehF9PDly6TleluZWZiXMtqagmeEI/sUKaaq0yrMrNrzfIfJYCD0hJx
8tBFZKwUVKaYQjB4Bo6Ij0V04dcP8m6NAfXNG0/MB56zj7WhMndOPP/tuIGmj1jV
jBopq/Pm8KUrzDVCmD+kQLT7hZTi0lapG7A/tE7bc2pGypP2hLR0i9/ckieXh4IL
YKdaNda4THjnA3bHC7/ELOWPinrmQlaxY1mW8hkLCHCQ/MG/SJKkVTY2fX62BscY
KGQPOwh9jNWIS8MAbNJafYBxVueje6Xe/e+OdwIDAQABAoIBAQDXOH4+u1eerVR5
m9gYmHM1LEqdqZ5QgGuoDC8KJY9buu7WcfmvltNpbq7afYI2GgkUuaX03tniq8lh
kkPqipzS9ObLtRtmgwCiAm1WYXey44YA0ag5EwvG87qtSnd1wI6bWqKL9A3lBLPD
B/U4BW8XVV7Z1IMd8So8fgsx+cwmqk2+CqMZsq2m7FBjrCt7fu0QMaOXvuUWka5D
K8F+yvAbD5UMXOqN9fjMuPzbtiOFXvmWtpC3L6wxZjmAPuiR9fNzwM4jCXSasMU8
PNy3x0Esn/ic7IP+1v1PEiXImLui4WPF74v8i6HB6FgxCBht6rCYyl8kYh68gJS6
HTmgf9+BAoGBAPk+ft+R4T/iGDZ0y1YhRNN5zxqtOEcLf1+5/s6W6b+FTRFrxYLL
A8ZMkn0Se8DfVjPald1gMSikE8vj8YJSHAMlstPiHiYQKsBMGbjksZLjBpSapkTy
iMjpk+SmVN3sZRU3jfrcMoPfAQx0hBrCUJdhZFV/aYLiuJ7GUub34gKzAoGBAOIU
mu5aht3MDjiR2mnNzEj2TY7YED7HNWOQJ0FzqLfl4jZTowMbnz8aFUWxkJmC3i1y
0Y8rWbxV7CwcXc9fOR8jRNx2Jn6eaA4A9+a02D1hNHJ6bQivy6yloQXCs4/wardY
EBHvepygs+v507khX146bl8PmUCIYpjVFaeOrRctAoGAehhLPmnP1eODyOld0ktp
086PzZmdP/A57ULHt5vl1ZQPNMF+d5vLtZA9ElfDl6/QIoapc1BzxFzb9b0ryZM/
das59uGFs0+oIZsl3pTpB/N+fb1kRdIpf4IsmI2CdVQgEEyumHzVohPUB63sKM+X
exCSfe90WFGH7v9oDQzRAlECgYBocSRx4JhVdqNLNvYz0sMBIegKiX5XwifD6yB3
eDsFWcn7VwADu4sB18bj/3fRs0d4r4ZoIZq/CuKkLiaYWmFFJUH2pw55iCyB66ia
iAktse5MxIoCbVQmWg3dX2kcofBq6t/hqUR3fzYfWbaZ2/T2zv+WItqlmVwTRr1O
PvdvsQKBgQDA0MQnQ7iV6Uvdm2AFHXlnwtXGelG0EsaNabF31LosHujMf92kUzaC
12NiuE0gEfuPemEw+MFQ/nfBiEDv23NgRz6frZvcvUYWr/xFs8AwPkV5pBFNwoa2
4lYPbMB/1RWf4zDKvDBKEpEGr3pCKBjrPszi0skn2DTCcpiqjH3XGg==
-----END RSA PRIVATE KEY-----
EOF
$ cluster/kubectl.sh delete -f enc-dply.yaml
deployment.apps "enc-nginx" deleted
$ cluster/kubectl.sh create -f enc-dply.yaml
deployment.apps/enc-nginx created
$ cluster/kubectl.sh get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
enc-nginx-7fb4578896-8grjd 1/1 Running 0 61s 10.88.0.102 127.0.0.1 <none> <none>
enc-nginx-7fb4578896-jjhz4 1/1 Running 0 61s 10.88.0.100 127.0.0.1 <none> <none>
enc-nginx-7fb4578896-jv9n5 1/1 Running 0 61s 10.88.0.101 127.0.0.1 <none> <none>
# test with curl
$ curl -v 10.88.0.92
< HTTP/1.1 200 OK
< Server: nginx/1.15.12
< Date: Thu, 27 Feb 2020 09:47:44 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
< Connection: keep-alive
< ETag: "5cb5d3c3-264"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
The main integration points for Encrypted Container Images
镜像加密工具
对于镜像加密工具,社区计划支持:docker CLI
(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714), containerd imgcrypt, skopeo以及buildah,目前已经实现的有containerd imgcrypt
和skopeo
镜像仓库
Docker Distribution >= v2.7.1
支持镜像加密OCI格式
Container Runtime
于Container Runtime,社区计划支持:containerd(+CRI-Containerd)以及cri-o(暂不支持Docker),目前已经实现的有containerd
(CRI-Containerd目前正在合并PR)以及cri-o
Kubernetes
目前社区计划对Kubernetes支持两种镜像加密模式:
- Node Key Model:将密钥放在Kubernetes工作节点上,以节点为解密粒度(CRIO已经实现,CRI-Containerd待合并PR)
- Multitenant Key Model:多租户模型,支持解密粒度为集群和用户(还未实现,based on this KEP)
另外,对于Node Key Model
模式,测试加密流程可行,并对性能评估如下:
- 1、目前社区对镜像加密这块并没有性能测试数据
- 2、利用对称加密算法(i.e. AES)加密镜像Layer,加密后数据长度不变——对镜像pull影响不大&解密速度快,耗费时间少
- 3、Memory would not be that bad since we are using a stream cipher
补充:目前社区对于Docker的支持计划参考如下(我的思考):应该是等到Docker底层代码切换到Containerd(moby/moby#38043),然后修改docker-shim支持镜像加密,所以目前是阻塞的状态
Conclusion
在容器安全涉及的内容中,目前还不十分成熟的部分是镜像加密
,镜像加密
在数据保密性要求较强的领域中会显得十分重要,例如:金融、银行以及国企等
本文首先介绍了镜像加密
涉及的OCI
规范,在此基础上对镜像加密
原理以及流程进行了说明,并利用containerd imgcrypt工具对加密流程进行了实操。而Kubernetes
对镜像加密特性支持Node Key Model
以及Multitenant Key Model
这两种使用模式,本文详细讲解了Kubernetes
基于Node Key Model
使用模式的原理&流程,并结合CRIO
演示了镜像加密Kubernetes-Native
的用法
虽然目前基于社区的工作可以在Kubernetes
上使用镜像加密特性,但是该特性的支持还并不完善,比如:对于加密工具,还需要支持docker CLI
(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714);对于容器运行时,还需要支持CRI-Containerd以及Docker
等;对于云原生生态系统,还需要更多地与镜像相关的项目(such as Notary, Clair and so on)进行合作;对于Kubernetes,还需要支持更多Key Models
Refs
- Encrypting container images with containerd imgcrypt!
- The main integration points for Encrypted Container Images
- current process
- How Encrypted Images brings about compliance in Kubernetes (via CRI-O)
- Support for Encrypted Images in Kubernetes
- Add Image Decryption KEP
- What’s the difference between runc, containerd, docker? Well, I asked myself the exact same question…
- Container Runtimes Part 1: An Introduction to Container Runtimes
- Encrypted container images for container image security at rest
- Running CRI-O on kubernetes cluster
- Encrypting container images with skopeo
- OCI Container Tools Guide
- OCI Image Implementations
- OCI Runtime Implementations
- OCI Image Manifest Specification
- CRI: the Container Runtime Interface