docker kubernetes 云原生容器启动加速-nydus

容器启动加速-nydus

1. 概述

1.1. 使用nydus需要考虑的问题1.2. 当前实践探索中遇到的问题 2. 容器集成nydus

2.1. contanierd 集成【核心】

2.1.1. 版本说明2.1.2. 概念说明2.1.3. 构建

2.1.3.1. 构建 buildkitd2.1.3.2. 安装nydus2.1.3.3. 启动buildkitd进程2.1.3.4. 构建镜像 2.1.4. 部署

2.1.4.1. 部署nydus-snapshotter2.1.4.2. 配置containerd支持nydus snapshotter 2.1.5. 使用nydus镜像启动containerd容器 2.2. kubernertes集成

2.2.1. 版本说明2.2.2. k8s集群准备

2.2.2.1. 将k8s cri从docker替换为containerd 2.2.3. 部署buildkit

2.2.3.1. 构建buildkit镜像2.2.3.2. 按需修改yaml并部署 2.2.4. 部署 nydus-snapshotter

2.2.4.1. 更改nydus镜像2.2.4.2. 修改configmap2.2.4.3. 重启nydus-snapshotter 2.2.5. 验证

2.2.5.1. 在pod中构建nydus格式的镜像2.2.5.2. 使用pod启动nydus镜像容器 2.3. docker 集成 3. CI集成 – skaffold集成nydus4. nydus镜像启动pod与ociv1镜像启动pod效果对比

4.1. wordpress镜像

4.1.1. oci v1镜像容器pulling image时间4.1.2. nydus镜像容器pulling image时间 5. 问题记录

5.1. 镜像无法构建5.2. containerd 命令速记5.3. 镜像无法下载5.4. 如何判断一个镜像是否是nydus镜像

5.4.1. 通过本地manifest判断5.4.2. 通过nydusify check检查 5.5. nydus snapshotter无法启动5.6. buildkit构建nydus镜像构建不成功5.7. dragonflyoss仓库的镜像无法 inspect5.8. nydus镜像与oci镜像manifest的区别5.9. 构建的镜像list时告警5.10. 问题 buildkit打的nydus镜像无法run ,run ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 没问题5.11. 运行oci-ref镜像导致nydusd进程z状态

5.11.1. 现象5.11.2. 解决方式 5.12. nydus镜像和oci镜像拉取镜像时间相同

5.12.1. nydus镜像启动 nydus-snapshotter日志5.12.2. oci镜像启动 nydus-snapshotter日志 6. 参考

1. 概述

Nydus是一个镜像加速器，主要通过容器启动时懒加载镜像提高启动速度, 大镜像可以秒级启动，后续的读写通过COW方式。以下是nydus的几个标签:

Nydus是蚂蚁、阿里云、字节等共建的开源容器镜像加速项目CNCF孵化的项目Dragonfly中的一个子项目新的镜像格式，不同于oci良好的生态，兼容containerd、k8s、harbor、buildkit – 从构建、存储、部署、到CI集成，有丰富的解决方案erofs+fscache+cachefiles 缓存方案: 使用内核缓存和只读文件系统进行io加速

cachefiles: 获取文件缓存fscache: 管理cachefiles的缓存内容erofs: 将缓存以文件系统接口方式提供访问

用Nydus对容器进行加速，当前可用，可以使用(k8s+)buildkit构建镜像，使用(k8s+)containerd进行发布, 使用skaffold进行CI(但路途坎坷，该文档内容主要为这部分)；未来可期，使用fscache+erofs进行底层存储加速。

1.1. 使用nydus需要考虑的问题

如果要落地Nydus，需要考虑如下几个问题:

当前k8s container runtime 如果是docker，则需要切换成containerd，切换需要注意

pod服务可能会中断监控日志，原来通过docker接口，现在需要通过containerd接口 – 这部分k8s可以屏蔽 CI如何处理原有的docker build方式和现在buildkit构建方式 containerd功能相对简单，要用它来进行构建、运行等操作需要多个组件配合才行，要使用起来需要做不少操作:

containerdnerdctlbuildkitdbuildctl …nydus-snapshotter 要使用nydus镜像的话需要 ⚠️性能问题，写时复制从远端拉取，例如从镜像仓库拉取的话需要考虑

镜像仓库本身的IO性能从pod到仓库的网络带宽 ⚠️容器化使用nydus-snapshotter有待改进现在nydus-snapshotter容器故障之后由于和容器启动需要nydus-snapshotter这样的循环依赖，所以无法自愈，需要

删除containerd中的nydus-snapshotter配置重启containerd重新配置containerd

1.2. 当前实践探索中遇到的问题

在实践探索中主要有如下问题:

涉及项目多，各项目版本不明确，遇到较多版本兼容问题文档分散，不易查阅实践类文档较少文档落后代码较多部分项目CI部分代码落后于主体代码

正在解决中的问题

构建出来的本地镜像缺少layer无法run

2. 容器集成nydus

2.1. contanierd 集成【核心】

2.1.1. 版本说明

组件版本备注内核5.18.2-1.el7.elrepo.x86_64containerdcontainerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1buildkitgithub.com/moby/buildkit 76aeaa55.m 76aeaa55ff4e1c575644bffbf0a43bea92215fbe.m从https://github.com/nydusaccelerator/buildkit.git构建nydus-imagev2.2.1containerd-nydus-grpcv0.8.0nerdctl1.3.1

2.1.2. 概念说明

nydus镜像: nydus格式的镜像，不同于现在的ociv1 ociv2，在使用时可进行懒加载，加快容器启动速度 nydus镜像构建

buildkitd: 构建nydus镜像的工具，需要增加nydus兼容配置, 一般以守护进程运行，客户端通过 buildctl 命令通过sock文件或tcp进行与服务端连接，也支持daemonlessnydus-image: nydus镜像工具，可以集成到buildkit中构建nydus格式的镜像，也可以单独使用进行nydus镜像构建、检查、导出等 nydus镜像运行

containerd: 容器运行时nydus-snapshotter: 是一个containerd插件，用来支持contanierd使用nydus格式的镜像1

buidlkit+nydus构建完镜像之后推送到仓库，可以是docker registry，也可以是harbor等运行的时候每启动一个nydus镜像的容器，containerd会调用 nydus-snapshotter 去查看本地有没有缓存，如果没有的话启动 nydusd 进程去拉取,如果有的话启动nydusd fuse进程作为文件服务器提供给容器挂载，这里是懒加载的关键，容器需要什么nydusd就会从仓库拉什么，启动对应的snapshotter挂载进程。

/usr/bin/nydusd fuse --thread-num 10 --config /var/lib/containerd-nydus/config/chmadgaihqqap9ldb1eg/config.json --bootstrap /var/lib/containerd-nydus/snapshots/833/fs/image/image.boot --mountpoint /var/lib/containerd-nydus/snapshots/833/mnt --apisock /var/lib/containerd-nydus/socket/chmadgaihqqap9ldb1eg/api.sock --log-level debug --log-file /var/lib/containerd-nydus/logs/chmadgaihqqap9ldb1eg/nydusd.log

架构图如下，这里用的是fuse文件系统, erofs的社区文档比较少，暂时先用fuse

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-jMiUFnzf-1686298186600)(null)]

如果用erofs

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vg8EBgO4-1686298186840)(null)]

2.1.3. 构建

2.1.3.1. 构建 buildkitd

官方社区仓库是 moby/buildkit, 这里用的nydusaccelerator的项目，是因为当前bug/新功能会先在nydusaccelerator发布，所以暂时先用这个项目

git clone https://github.com/nydusaccelerator/buildkit.git

cd buildkit

go build -tags=nydus -o ./bin/buildkitd ./cmd/buildkitd

mv ./bin/buildkitd /usr/bin/

chmod +x /usr/bin/buildkitd

2.1.3.2. 安装nydus

wget https://github.com/dragonflyoss/image-service/releases/download/v2.2.1/nydus-static-v2.2.1-linux-amd64.tgz

tar zxf nydus-static-v2.2.1-linux-amd64.tgz

cd nydus-static

sudo install -D -m 755 nydusd nydus-image nydusify nydusctl nydus-overlayfs /usr/bin

2.1.3.3. 启动buildkitd进程

buildkit 配置2

cat > /etc/buildkit/buildkitd.toml <

debug = false

[worker.oci]

enabled = false

[worker.containerd]

enabled = true

# 构建成功的之后的镜像存储位置

namespace = "default"

address = "/run/containerd/containerd.sock"

enabled = true

platforms = [ "linux/amd64", "linux/arm64" ]

gc = true

# gckeepstorage sets storage limit for default gc profile, in MB.

gckeepstorage = 9000

EOF

cat > /etc/systemd/system/buildkitd.service << EOF

[Unit]

Description=BuildKit

Documentation=https://github.com/moby/buildkit

[Service]

Environment="NYDUS_BUILDER=/usr/bin/nydus-image"

ExecStart=/usr/bin/buildkitd --config /etc/buildkit/buildkitd.toml

[Install]

WantedBy=multi-user.target

EOF

systemctl enable buildkitd

systemctl start buildkitd

2.1.3.4. 构建镜像

构建一个简单的hello world镜像

hello world程序

mkdir golang-helloworld

编辑 main.go

package main

import "fmt"

func main() {

fmt.Println("Hello, world!")

}

构建可执行文件

go build -o hello main.go

Dockerfile

编辑dockerfile

FROM ubuntu:20.04

COPY hello /

ENTRYPOINT [ "/hello" ]

构建

这里会构建一个nydus格式的镜像名称为reg.harbor.com/hello/hello-k8s:v0.0.4, push 为 true 则构建完成之后推送远端仓库

buildctl build --progress=plain \

--frontend=dockerfile.v0 \

--local context=. \

--local dockerfile=. \

--output type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.4,compression=nydus,oci-mediatypes=true,push=true

校验

这里我们构建出来了nydus镜像，可以校验一下

nydusify check --target reg.harbor.com/hello/hello-k8s:v0.0.4

INFO[2023-05-24T10:59:56+08:00] Parsing image reg.harbor.com/hello/hello-k8s:v0.0.4

INFO[2023-05-24T10:59:56+08:00] Dumping OCI and Nydus manifests to ./output

...

INFO[2023-05-24T10:59:57+08:00] Verified Nydus image reg.harbor.com/hello/hello-k8s:v0.0.4

这里check的是远端镜像，一切符合预期,是nydus镜像3

cat output/nydus_manifest.json

{

"schemaVersion": 2,

"mediaType": "application/vnd.oci.image.manifest.v1+json",

"config": {

"mediaType": "application/vnd.oci.image.config.v1+json",

"digest": "sha256:3121880b13bcaf21d0748cc35139769cb353bdd8429e541da489d1833c0c9ea7",

"size": 6940

"layers": [

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a",

"size": 30779473,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

...

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab",

"size": 1067099,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

{

"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",

"digest": "sha256:4c73ce0e2ddc066d200bc119cc2b6f39938fd136b93fb8d47a590788cc91c91c",

"size": 2258453,

"annotations": {

"containerd.io/snapshot/nydus-blob-ids": "[\"e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a\",\"15596146dcdb421faaf9881e63baa87731603f412d1f05aa78e2d8f5bb78f1d5\",\"68fb6d505c4733f3c39645876ab9ed30dbf46056b36630be2dafe48634af1e01\",\"32a3c81c8fb5b12c68d74e01b29cd02463d8e85c0296ac3e30874f0f38412ec9\",\"89fe8a5a29e872cb9afbdeffeb9307498e67f42a0d3fdeca754b7a55b8849083\",\"43f9d24cfd0900b5ccbc7030559df305d237932f5383f592128d165c723ca553\",\"5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab\"]",

"containerd.io/snapshot/nydus-bootstrap": "true",

"containerd.io/snapshot/nydus-fs-version": ""

}

]

}

本地镜像暂时无法使用,使用会报错(这里是部署了nydus snapshotter之后)

nerdctl run --rm reg.harbor.com/hello/hello-k8s:v0.0.4 --snapshotter=nydus

INFO[0000] apply failure, attempting cleanup error="failed to extract layer sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a: failed to get reader from content store: content digest sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a: not found" key="extract-504360049-gvw0 sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a"

FATA[0000] failed to extract layer sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a: failed to get reader from content store: content digest sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a: not found

这里应该应该是nydus+fuse的bug，解决方法为在构建完成之后把本地镜像删除掉, 使用时通过远端拉取

nerdctl rmi reg.harbor.com/hello/hello-k8s:v0.0.4

本地镜像构建出来的manifest和远端一致，这里不能用是因为存储的时候少存储了snapshotter，具体问题待社区确认

2.1.4. 部署

2.1.4.1. 部署nydus-snapshotter

安装 containerd-nydus-grpc

wget https://github.com/containerd/nydus-snapshotter/releases/download/v0.8.0/nydus-snapshotter-v0.8.0-x86_64.tgz

tar zxf nydus-snapshotter-v0.8.0-x86_64.tgz

install -D -m 755 nydus-snapshotter/containerd-nydus-grpc /usr/bin

nydus-fusedev 配置

# nydus blob 缓存

mkdir /opt/nydus/cache

cat > /etc/nydus/nydusd-config.fusedev.json <

{

"device": {

"backend": {

"type": "registry",

"config": {

"scheme": "https",

"skip_verify": true,

"timeout": 5,

"connect_timeout": 5,

"retry_limit": 4

}

"cache": {

"type": "blobcache",

"config": {

"work_dir": "/opt/nydus/cache"

}

"mode": "direct",

"digest_validate": false,

"iostats_files": false,

"enable_xattr": true,

"fs_prefetch": {

"enable": true,

"threads_count": 4

}

EOF

nydus snapshotter 主进程配置4

cat /etc/nydus/config.toml

version = 1

address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

daemon_mode = "multiple"

cleanup_on_close = false

[system]

enable = true

address = "/run/containerd-nydus/system.sock"

[system.debug]

daemon_cpu_profile_duration_secs = 5

pprof_address = ""

[daemon]

nydusd_path = "/usr/bin/nydusd"

nydusimage_path = "/usr/bin/nydus-image"

# fusedev or fscache

fs_driver = "fusedev"

# How to process when daemon dies: "none", "restart" or "failover"

recover_policy = "restart"

# Specify a configuration file for nydusd

nydusd_config = "/etc/nydus/nydusd-config.fusedev.json"

# The fuse or fscache IO working threads started by nydusd

threads_number = 10

[log]

# Print logs to stdout rather than logging files

log_to_stdout = false

level = "debug"

log_rotation_compress = true

log_rotation_local_time = true

log_rotation_max_age = 7

log_rotation_max_backups = 5

log_rotation_max_size = 1

[metrics]

# Enable by assigning an address, empty indicates metrics server is disabled

address = ":9110"

[remote]

convert_vpc_registry = false

[remote.mirrors_config]

dir = ""

[remote.auth]

enable_kubeconfig_keychain = false

kubeconfig_path = ""

enable_cri_keychain = false

image_service_address = ""

[snapshot]

enable_nydus_overlayfs = false

sync_remove = false

[cache_manager]

disable = false

gc_period = "24h"

cache_dir = "/opt/nydus/cache/"

[image]

public_key_file = ""

validate_signature = false

[experimental]

enable_stargz = false

enable_referrer_detect = false

cat > /etc/systemd/system/nydus-snapshotter.service <

[Unit]

Description=nydus snapshotter

After=network.target

Before=containerd.service

[Service]

Type=simple

Environment=HOME=/root

ExecStart=/usr/local/bin/containerd-nydus-grpc --config /etc/nydus/config.toml

Restart=always

RestartSec=1

KillMode=process

OOMScoreAdjust=-999

StandardOutput=journal

StandardError=journal

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable nydus-snapshotter

systemctl start nydus-snapshotter

2.1.4.2. 配置containerd支持nydus snapshotter

编辑containerd配置 /etc/containerd/config.toml

version = 2

state = "/run/containerd"

[grpc]

address = "/run/containerd/containerd.sock"

uid = 0

gid = 0

max_recv_message_size = 16777216

max_send_message_size = 16777216

[timeouts]

"io.containerd.timeout.shim.cleanup" = "5s"

"io.containerd.timeout.shim.load" = "5s"

"io.containerd.timeout.shim.shutdown" = "3s"

"io.containerd.timeout.task.state" = "2s"

[plugins]

[plugins."io.containerd.grpc.v1.cri"]

# k8s原来用的container runtime是docker，则需要配置pause镜像，否则runtime改编之后k8s原有pod无法启动

sandbox_image = "reg.harbor.com/kubernetes/pause:3.6"

[plugins."io.containerd.grpc.v1.cri".cni]

# cni插件路径

bin_dir = "/opt/cni/bin"

conf_dir = "/etc/cni/net.d"

max_conf_num = 1

conf_template = ""

[plugins."io.containerd.grpc.v1.cri".registry]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]

endpoint = ["https://registry-1.docker.io"]

# Nydus插件主要配置

[proxy_plugins]

[proxy_plugins.nydus]

type = "snapshot"

address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]

snapshotter = "nydus"

disable_snapshot_annotations = false

discard_unpacked_layers = false

修改完毕重启containerd

systemctl restart containerd

2.1.5. 使用nydus镜像启动containerd容器

containerd也有类似docker的命令行 nerdctl

安装 nerdctl

wget https://github.com/containerd/nerdctl/releases/download/v1.3.1/nerdctl-1.3.1-linux-amd64.tar.gz

tar zxvf nerdctl-1.3.1-linux-amd64.tar.gz

install -D -m 755 nerdctl /usr/bin/

配置 nerdctl

默认使用 nydus-snapshotter

cat > /etc/nerdctl/nerdctl.toml <

debug = false

debug_full = false

address = "unix:///var/run/containerd/containerd.sock"

namespace = "default"

snapshotter = "nydus"

experimental = true

EOF

使用构建阶段产生的镜像运行容器

先删除本地镜像 nerdctl rmi reg.harbor.com/hello/hello-k8s:v0.0.4

nerdctl run --rm reg.harbor.com/hello/hello-k8s:v0.0.4

# 输出

reg.harbor.com/hello/hello-k8s:v0.0.4: resolved |++++++++++++++++++++++++++++++++++++++|

manifest-sha256:c6ab4a56b80d503312322bda42fc31b038d6f6989d8d9e801c991787b8301132: exists |++++++++++++++++++++++++++++++++++++++|

config-sha256:3121880b13bcaf21d0748cc35139769cb353bdd8429e541da489d1833c0c9ea7: exists |++++++++++++++++++++++++++++++++++++++|

layer-sha256:4c73ce0e2ddc066d200bc119cc2b6f39938fd136b93fb8d47a590788cc91c91c: done |++++++++++++++++++++++++++++++++++++++|

elapsed: 0.6 s total: 2.2 Mi (3.6 MiB/s)

Hello, world!

使用 nerdctl 命令时可能会遇到 failed to call cni.Setup: plugin type=\"bridge\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]" cni不支持的问题，需要升级cni

https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-arm64-v1.1.1.tgz

tar zxf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin

升级完毕即可用

到这里使用nydus加速containerd容器启动，基本完成。使用erofs待探索

2.2. kubernertes集成

2.2.1. 版本说明

组件版本备注k8sv1.22.10containerdcontainerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1buildkitdgithub.com/moby/buildkit 76aeaa55.m 76aeaa55ff4e1c575644bffbf0a43bea92215fbe.m从https://github.com/nydusaccelerator/buildkit.git构建nydus-snapshotterv0.6.1通过helm部署5,生产用的话需要更新下

2.2.2. k8s集群准备

首先需要一个 container runtime为containerd的集群

kubectl describe nodes |grep -i runtime

Container Runtime Version: docker://20.10.8

如果是docker，则需要更改为containerd

2.2.2.1. 将k8s cri从docker替换为containerd

此处操作比较粗糙，生产切换请慎重

更改 kubelet 配置

编辑/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

增加/在KUBELET_EXTRA_ARGS追加如内容

Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"

创建 containerd 配置

新建 ``

version = 2

state = "/run/containerd"

[plugins]

[plugins."io.containerd.grpc.v1.cri"]

sandbox_image = "reg.harbor.com/kubernetes/pause:3.6"

You have new mail in /var/spool/mail/root

重启containerd kubelet服务

systemctl daemon-reload

systemctl restart containerd kubelet

如果 kubelet 启动失败，请查看日志做对应处理 journalctl -xu kubelet -f

启动成功之后检查如下输出之后说明切换成功

kubectl describe nodes |grep -i runtime

# 三个节点

Container Runtime Version: containerd://1.6.6

2.2.3. 部署buildkit

2.2.3.1. 构建buildkit镜像

社区的镜像不支持nydus，需要构建镜像

git clone https://github.com/nydusaccelerator/buildkit.git

cd buildkit

修改 Dockerfile

diff --git a/Dockerfile b/Dockerfile

index 8869138b..b006e584 100644

--- a/Dockerfile

+++ b/Dockerfile

@@ -97,7 +97,7 @@ ARG TARGETPLATFORM

RUN --mount=target=. --mount=target=/root/.cache,type=cache \

--mount=target=/go/pkg/mod,type=cache \

--mount=source=/tmp/.ldflags,target=/tmp/.ldflags,from=buildkit-version \

- CGO_ENABLED=0 xx-go build -ldflags "$(cat /tmp/.ldflags) -extldflags '-static'" -tags "osusergo netgo static_build seccomp ${BUILDKITD_TAGS}" -o /usr/bin/buildkitd ./cmd/buildkitd && \

+ CGO_ENABLED=0 xx-go build -ldflags "$(cat /tmp/.ldflags) -extldflags '-static'" -tags "osusergo netgo nydus static_build seccomp ${BUILDKITD_TAGS}" -o /usr/bin/buildkitd ./cmd/buildkitd && \

xx-verify --static /usr/bin/buildkitd

FROM scratch AS binaries-linux-helper

@@ -176,6 +176,7 @@ ARG ROOTLESSKIT_VERSION

RUN git clone https://github.com/rootless-containers/rootlesskit.git /go/src/github.com/rootless-containers/rootlesskit

WORKDIR /go/src/github.com/rootless-containers/rootlesskit

ARG TARGETPLATFORM

+ENV GOPROXY "https://goproxy.cn,direct"

RUN --mount=target=/root/.cache,type=cache \

git checkout -q "$ROOTLESSKIT_VERSION" && \

CGO_ENABLED=0 xx-go build -o /rootlesskit ./cmd/rootlesskit && \

@@ -203,6 +204,7 @@ RUN mkdir -p /out/nydus-static && tar xzvf nydus-static-$NYDUS_VERSION-$TARGETOS

FROM buildkit-export AS buildkit-linux

COPY --link --from=binaries / /usr/bin/

+COPY --from=nydus /out/nydus-static/* /usr/bin/

ENTRYPOINT ["buildkitd"]

FROM binaries AS buildkit-darwin

主要有三处修改:

编译增加nydus支持 – 必须增加goproxy配置 – 国内拉依赖包会失败，需要配置goproxy将nydus-image打入镜像，这步经测试可有可无

make build

会构建出两个镜像，我们只使用非rootless的 moby/buildkit:local

2.2.3.2. 按需修改yaml并部署

git clone https://github.com/nydusaccelerator/buildkit.git

cd buildkit

这里使用deployment+service.privileged 6

编辑 deployment+service.privileged.yaml，去掉了tls配置，增加了NYDUS_BUILDER环境变量，最终如下

镜像 shoery/buildkit:76aeaa55

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: buildkitd

spec:

replicas: 1

selector:

matchLabels:

app: buildkitd

template:

metadata:

labels:

app: buildkitd

spec:

containers:

- name: buildkitd

env:

- name: NYDUS_BUILDER

value: /usr/bin/nydus-image

image: shoery/buildkit:76aeaa55

args:

- --addr

- unix:///run/buildkit/buildkitd.sock

- --addr

- tcp://0.0.0.0:1234

# the probe below will only work after Release v0.6.3

readinessProbe:

exec:

command:

- buildctl

- debug

- workers

initialDelaySeconds: 5

periodSeconds: 30

# the probe below will only work after Release v0.6.3

livenessProbe:

exec:

command:

- buildctl

- debug

- workers

initialDelaySeconds: 5

periodSeconds: 30

securityContext:

privileged: true

ports:

- containerPort: 1234

---

apiVersion: v1

kind: Service

metadata:

labels:

app: buildkitd

spec:

ports:

- port: 1234

protocol: TCP

selector:

app: buildkitd

部署

kubectl apply -f examples/kubernetes/deployment+service.privileged.yaml

构建镜像时仅需客户端有 buildctl 即可, 通过 --addr tcp://0.0.0.0:1234 连接到 buildkitd

2.2.4. 部署 nydus-snapshotter

run_nydus_in_kubernetes

这里注意用 v0.8.2 以上的镜像版本

这里没太多修改的地方, 两个地方需要修改，其他按文档部署即可

git clone https://github.com/dragonflyoss/helm-charts.git

cd helm-charts

2.2.4.1. 更改nydus镜像

新建 config-nydus.yaml

image: ghcr.io/containerd/nydus-snapshotter

tag: v0.8.2

pullPolicy: IfNotPresent

hostNetwork: true

dragonfly:

enable: false

containerRuntime:

containerd:

enable: true

2.2.4.2. 修改configmap

如果不增加nydus.toml配置，懒加载不会生效

编辑 charts/nydus-snapshotter/templates/nydus-snapshotter/snapshotter-configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

labels:

app: {{ template "nydus-snapshotter.fullname" . }}

chart: {{ .Chart.Name }}-{{ .Chart.Version }}

release: {{ .Release.Name }}

heritage: {{ .Release.Service }}

component: {{ .Values.name }}

data:

config.json: |-

{

"device": {

"backend": {

"type": "registry",

"config": {

"mirrors": {{ mustToJson .Values.dragonfly.mirrorConfig }},

"timeout": 5,

"connect_timeout": 5,

"retry_limit": 2

}

"cache": {

"type": "blobcache",

"config": {

"work_dir": "/var/lib/nydus/cache/"

}

"mode": "direct",

"digest_validate": false,

"iostats_files": false,

"enable_xattr": true,

"fs_prefetch": {

"enable": true,

"threads_count": 8,

"merging_size": 1048576,

"prefetch_all": true

}

nydus.toml: |-

version = 1

address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

daemon_mode = "multiple"

cleanup_on_close = false

[system]

enable = true

address = "/run/containerd-nydus/system.sock"

[system.debug]

daemon_cpu_profile_duration_secs = 5

pprof_address = ""

[daemon]

nydusd_path = "/usr/local/bin/nydusd"

nydusimage_path = "/usr/local/bin/nydus-image"

fs_driver = "fusedev"

recover_policy = "restart"

nydusd_config = "/etc/nydus/config.json"

threads_number = 10

[log]

log_to_stdout = false

level = "debug"

log_rotation_compress = true

log_rotation_local_time = true

log_rotation_max_age = 7

log_rotation_max_backups = 5

log_rotation_max_size = 1

[metrics]

address = ":9110"

[remote]

convert_vpc_registry = false

[remote.mirrors_config]

dir = ""

[remote.auth]

enable_kubeconfig_keychain = false

kubeconfig_path = ""

enable_cri_keychain = false

image_service_address = ""

[snapshot]

enable_nydus_overlayfs = false

sync_remove = false

[cache_manager]

disable = false

gc_period = "24h"

[image]

public_key_file = ""

validate_signature = false

[experimental]

enable_stargz = false

enable_referrer_detect = false

安装

helm install --wait --timeout 10m --dependency-update \

--create-namespace --namespace nydus-system \

-f config-nydus.yaml \

nydus-snapshotter charts/nydus-snapshotter

2.2.4.3. 重启nydus-snapshotter

后续维护中可能会有重启 nydus-snapshotter 的需求

由于修改containerd配置使用nyduys snapshotter之后, containerd和snapshotter会互相依赖，重启nydus-snapshotter容器会起不来，所以，每次重启都先把 /etc/containerd/config.toml 中如下部分去掉

[plugins."io.containerd.grpc.v1.cri".containerd]

discard_unpacked_layers = false

disable_snapshot_annotations = false

snapshotter = "nydus"

[proxy_plugins]

[proxy_plugins.nydus]

type = "snapshot"

address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

重启 containerd 重新创建 nydus-snapshotter pod

2.2.5. 验证

验证点: 在pod中构建nydus格式的镜像，使用pod启动nydus镜像容器

2.2.5.1. 在pod中构建nydus格式的镜像

要挂载 /root/.docker, 否则push镜像可能会没权限

挂载golang-helloworld的目的是将 hello 和 Dockerfile挂载进去

创建 build-nydus-image.yaml

apiVersion: v1

kind: Pod

metadata:

spec:

restartPolicy: Never

containers:

- name: build-container

image: buildkit:local

command: ["buildctl", "--addr", "tcp://buildkitd.default:1234", "build", "--progress=plain", "--frontend=dockerfile.v0", "--local", "context=.", "--local", "dockerfile=.", "--output", "type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.6,compression=nydus,oci-mediatypes=true,push=true"]

volumeMounts:

- name: host-path-volume

mountPath: /opt/golang-helloworld

- name: docker-config-volume

mountPath: /root/.docker

workingDir: /opt/golang-helloworld

volumes:

- name: host-path-volume

hostPath:

path: /home/elrond/nydus/golang-helloworld

- name: docker-config-volume

hostPath:

path: /root/.docker

命令说明

buildctl \

# buildkitd的svc:port

--addr tcp://default.buildkitd:1234 \

build \

# 输出的详细程度

--progress=plain \

# build frontend 类型

--frontend=dockerfile.v0 \

# 指定本地context路径

--local context=. \

# 制定本地dockerfile路径

--local dockerfile=. \

# 指定dockerfile文件，如果为空，默认值为Dockerfile

--opt filename=hello.dockerfile

# 镜像tag与类型、是否推送到远端

--output type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.6,compression=nydus,oci-mediatypes=true,push=true

开始构建

kubectl apply -f build-nydus-image.yaml

构建成功

k get po

NAME READY STATUS RESTARTS AGE

build-pod 0/1 Completed 0 23s

2.2.5.2. 使用pod启动nydus镜像容器

创建 nydus-pod.yaml

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- name: hello

image: reg.harbor.com/hello/hello-k8s:v0.0.6

imagePullPolicy: Always

command: ["sh", "-c"]

args:

- tail -f /dev/null

运行

kubectl apply -f nydus-pod.yaml

这里可能会因为构建的镜像不正常，导致pod起不来，可以使用标准镜像 ghcr.io/dragonflyoss/image-service/nginx:nydus-latest 做测试

到这里在k8s上完成了nydus镜像的构建，和nydus镜像格式容器的启动

2.3. docker 集成

参照docker-env-setup

docker-nydus-graphdriver 尚在 Experimental 状态，且k8s在转向containerd，云原生社区对docker支持变弱，所以后续不考虑使用docker

3. CI集成 – skaffold集成nydus

编辑 skaffold.yaml

apiVersion: skaffold/v4beta4

kind: Config

build:

tagPolicy:

envTemplate:

artifacts:

- image: reg.harbor.com/hello/hello-k8s

custom:

buildCommand: buildctl --addr tcp://10.233.98.169:1234 build --progress=plain --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=reg.harbor.com/hello/hello-k8s:"{{.FOO}}",compression=nydus,oci-mediatypes=true,push=true --opt filename=hello.dockerfile

local:

push: true

入参:

–addr tcp://10.233.98.169 容器化的话，这个是svc的host和port–local dockerfile=.: dockerfile路径–output name=reg.harbor.com/hello/hello-k8s:“{{.FOO}}”: 镜像tag–opt filename: dockerfile文件名

其他都是默认值

4. nydus镜像启动pod与ociv1镜像启动pod效果对比

镜像ociv1拉取时间nydus拉取时间wordpress12.26946259s1.879414943s

4.1. wordpress镜像

先把镜像拉下来,推送到自己的仓库，然后转化一下推送到自己的仓库，用镜像启动k8s pod

docker pull wordpress:latest

docker tag wordpress:latest reg.harbor.com/hello/wordpress:latest

nydusify -D convert --nydus-image /usr/local/bin/nydus-image \

--source reg.harbor.com/hello/wordpress:latest \

--target reg.harbor.com/hello/wordpress:latest-nydus

4.1.1. oci v1镜像容器pulling image时间

新建 wordpress-ociv1.yaml

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- name: wordpress

image: reg.harbor.com/hello/wordpress:latest

imagePullPolicy: Always

command:

- sleep

- "36000000000"

resources:

limits:

cpu: "0.5"

memory: "256Mi"

kubectl apply -f wordpress-ociv1.yaml

kubectl describe po wordpress-ociv1

Normal Pulling 3m20s kubelet Pulling image "reg.harbor.com/hello/wordpress:latest"

Normal Pulled 3m7s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest" in 12.26946259s

4.1.2. nydus镜像容器pulling image时间

新建 wordpress-nydus.yaml

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- name: wordpress

image: reg.harbor.com/hello/wordpress:latest-nydus

imagePullPolicy: Always

command:

- sleep

- "36000000000"

resources:

limits:

cpu: "0.5"

memory: "256Mi"

Normal Pulling 40s kubelet Pulling image "reg.harbor.com/hello/wordpress:latest-nydus"

Normal Pulled 39s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus" in 1.879414943s

5. 问题记录

5.1. 镜像无法构建

git clone https://github.com/nydusaccelerator/buildkit.git

cd buildkit

go build -tags=nydus -o ./bin/buildkitd ./cmd/buildkitd

mv ./bin/buildkitd /usr/bin/

chmod +x /usr/bin/buildkitd

原生的buildkit不支持nydus格式构建，需要重新编译支持官方的buildkit对大镜像构建有bug，需要用nydus官方上游的仓库

5.2. containerd 命令速记

container与task7

主要用nerdctl工具

# 启动容器

ctr run -d docker.io/library/nginx:alpine my-container

# 容器列表

ctr -n default c ls

# task列表

ctr -n default task ls

# 删除容器

ctr -n default delete xxxxx

# 容器交互式命令

ctr task exec --tty --exec-id my-container my-container bash

# 镜像列表

ctr -n default images ls

# 获取contanierd镜像的manifest 先找到RepoDigests sha256

nerdctl inspect d88ba62a3cbb

cat /opt/docker/io.containerd.content.v1.content/blobs/sha256/d88ba62a3cbb999055927cbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3

# 转换镜像并推送到仓库

nydusify -D convert --nydus-image /usr/local/bin/nydus-image --source reg.harbor.com:443/library/golang@sha256:4b5cc6ca083856af8f58ab332f279b4401e1e46cc6d999957d5f88c563df24a7 --target reg.harbor.com:443/library/golang:1.18.7-nydus

nydusify -D convert --nydus-image /usr/local/bin/nydus-image --source ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 --target reg.harbor.com/hello/ubuntu:nydus-nightly-v6

5.3. 镜像无法下载

nerdctl pull ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5

ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5: resolved |++++++++++++++++++++++++++++++++++++++|

index-sha256:cb13d8b2c3b275fd392991b610e31462ab5dc588d4735b0c469ff867f86f617f: exists |++++++++++++++++++++++++++++++++++++++|

manifest-sha256:d9ea5e7108f5405da73ae48972441da2ab976da640264c4c06e07a473ebb86c3: exists |++++++++++++++++++++++++++++++++++++++|

config-sha256:d2197022013289d923e5b5aa9c9e22ec60bcdfa236948a02d24106b525f6171e: exists |++++++++++++++++++++++++++++++++++++++|

layer-sha256:f264de8a417d8927c431ef2a74286e24d716c3447a96ef7c7bd95d3af4ab3331: exists |++++++++++++++++++++++++++++++++++++++|

layer-sha256:c279d170bbf5c24d3b0b80bc1b49d147098d0f9f3f62cb010e98295b8d0e1f7c: downloading |+++++++++++++++++++++++++++++++++++++-| 31.0 MiB/31.1 MiB

docker.io/library/alpine:latest: resolved |++++++++++++++++++++++++++++++++++++++|

index-sha256:02bb6f428431fbc2809c5d1b41eab5a68350194fb508869a33cb1af4444c9b11: done |++++++++++++++++++++++++++++++++++++++|

manifest-sha256:c0669ef34cdc14332c0f1ab0c2c01acb91d96014b172f1a76f3a39e63d1f0bda: done |++++++++++++++++++++++++++++++++++++++|

config-sha256:5e2b554c1c45d22c9d1aa836828828e320a26011b76c08631ac896cbc3625e3e: done |++++++++++++++++++++++++++++++++++++++|

layer-sha256:8a49fdb3b6a5ff2bd8ec6a86c05b2922a0f7454579ecc07637e94dfd1d0639b6: done |++++++++++++++++++++++++++++++++++++++|

elapsed: 44.9s total: 3.2 Mi (73.9 KiB/s)

FATA[0045] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-05-15T21:27:17+08:00" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"

Failed to write to log, write /var/lib/nerdctl/1935db59/containers/default/286be294817095497aef32271dac13b13e02756166a010917420039060f9aa9c/oci-hook.createRuntime.log: file already closed: unknown

解决升级cni到v1.1.1

5.4. 如何判断一个镜像是否是nydus镜像

如何判断一个镜像是否是nydus镜像8

5.4.1. 通过本地manifest判断

如果镜像在containerd上

nerdctl inspect d88ba62a3cbb

# 找到RepoDigests sha256 d88ba62a3cbb999055927cbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3

[

{

"Id": "sha256:13afb147bdb4382b43a833b646765dca8bc6fca92cc2a60b02e4fc718b356d97",

"RepoTags": [

"reg.harbor.com/hello/helloworld:v0.0.1"

"RepoDigests": [

"reg.harbor.com/hello/helloworld@sha256:d88ba62a3cbb999055927cbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3"

"Comment": "buildkit.dockerfile.v0",

"Created": "2023-05-16T20:28:18.501558103+08:00",

"Author": "",

"Config": {

"AttachStdin": false,

"Env": [

"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

"Entrypoint": [

"/hello"

"Labels": {

"org.opencontainers.image.ref.name": "ubuntu",

"org.opencontainers.image.version": "22.04"

}

"Architecture": "amd64",

"Os": "linux",

"Size": 79822848,

"RootFS": {

"Type": "layers",

"Layers": [

"sha256:f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca",

"sha256:95fc9a6f9c1a1ef74f6545d70c96011577f05741079cd8793aa134f4b1f51006",

"sha256:719749732676f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24",

"sha256:ed593dc031518310cd5ea3a5b20bfb2c73f6108badffb30b080de9edc251476d"

]

"Metadata": {

"LastTagTime": "0001-01-01T00:00:00Z"

}

]

查看manifest

# containerd的默认是/var/lib/containerd

cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d88ba62a3cbb999055927cbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3

mediaType是nydus就是nydus镜像

{

"mediaType": "application/vnd.oci.image.manifest.v1+json",

"schemaVersion": 2,

"config": {

"mediaType": "application/vnd.oci.image.config.v1+json",

"digest": "sha256:13afb147bdb4382b43a833b646765dca8bc6fca92cc2a60b02e4fc718b356d97",

"size": 2137

"layers": [

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca",

"size": 32571260,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:95fc9a6f9c1a1ef74f6545d70c96011577f05741079cd8793aa134f4b1f51006",

"size": 295017,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:719749732676f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24",

"size": 1067099,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",

"digest": "sha256:bb55b253c36e5a008051aad37f2554a65ec6661476a08666d76e5ee1b84dbf4c",

"size": 297871,

"annotations": {

"containerd.io/snapshot/nydus-blob-ids": "[\"f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca\",\"95fc9a6f9c1a1ef74f6545d70c96011577f05741079cd8793aa134f4b1f51006\",\"719749732676f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24\"]",

"containerd.io/snapshot/nydus-bootstrap": "true",

"containerd.io/snapshot/nydus-fs-version": ""

}

]

}

5.4.2. 通过nydusify check检查

nydusify check --target reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5

INFO[2023-05-18T10:46:09+08:00] Parsing image reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5

INFO[2023-05-18T10:46:11+08:00] Dumping OCI and Nydus manifests to ./output

INFO[2023-05-18T10:46:11+08:00] Pulling Nydus bootstrap to output/nydus_bootstrap

INFO[2023-05-18T10:46:11+08:00] Checking Nydus manifest

INFO[2023-05-18T10:46:11+08:00] Checking Nydus bootstrap

RAFS filesystem metadata is valid, referenced data blobs:

0: f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca, compressed data size 0x1e5beec, compressed file size 0x1e5c0ec, uncompressed file size 0x49b715a, chunks: 0x9d4, features: cap_toc tar-header

1: ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129, compressed data size 0x459da, compressed file size 0x45bda, uncompressed file size 0xb3c90, chunks: 0x1, features: cap_toc tar-header

2: 5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab, compressed data size 0x102213, compressed file size 0x102413, uncompressed file size 0x1ad499, chunks: 0x2, features: cap_toc tar-header

INFO[2023-05-18T10:46:11+08:00] Verified Nydus image reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5

查看nydus manifest

cat output/nydus_manifest.json

{

"schemaVersion": 2,

"mediaType": "application/vnd.oci.image.manifest.v1+json",

"config": {

"mediaType": "application/vnd.oci.image.config.v1+json",

"digest": "sha256:f259d1eb0bf769d60aac8c78e4811a8eeefd811a27c24f6db6c36f72ac5c2324",

"size": 2137

"layers": [

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca",

"size": 32571260,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129",

"size": 295018,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab",

"size": 1067099,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",

"digest": "sha256:e26163305101a9bc6d05ad80945a8d360d99a48b0d3ef070cf0c2137655b6115",

"size": 297868,

"annotations": {

"containerd.io/snapshot/nydus-blob-ids": "[\"f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca\",\"ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129\",\"5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab\"]",

"containerd.io/snapshot/nydus-bootstrap": "true",

"containerd.io/snapshot/nydus-fs-version": ""

}

]

}

5.5. nydus snapshotter无法启动

可能是有另一个 nydus-snapshotter进程在启动 ps -ef|grep containerd-nydus-grpc

/usr/local/bin/containerd-nydus-grpc --root /opt/docker/io.containerd.snapshotter.v1.nydus --config-path /etc/nydus/nydusd-config.fusedev.json --config /etc/nydus/config.toml --log-to-stdout --log-level trace

INFO[2023-05-17T15:08:33.047605411+08:00] Start nydus-snapshotter. PID 25641 Version v0.8.0 FsDriver fusedev DaemonMode multiple

FATA[2023-05-17T15:08:37.018373244+08:00] failed to start nydus-snapshotter error="failed to initialize snapshotter: create database: timeout"ß

5.6. buildkit构建nydus镜像构建不成功

buildctl build --progress=plain --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=reg.harbor.com/hello/helloworld:v0.0.4-nydus-from-golang,compression=nydus,oci-mediatypes=true,push=true

#6 [2/2] COPY hello /

#6 ERROR: mount callback failed on /run/user/0/containerd-mount2534958279: apply nydus blob: unpack nydus blob: unpack nydus tar: unpack bootstrap from nydus: parse tar header: rpc error: code = OutOfRange desc = read past object length 54797046 bytes

------

> [2/2] COPY hello /:

------

Dockerfile:2

--------------------

1 | FROM reg.harbor.com/library/golang:1.18.7-nydus

2 | >>> COPY hello /

3 | ENTRYPOINT [ "/hello" ]

4 |

--------------------

error: failed to solve: OutOfRange: failed to compute cache key: mount callback failed on /run/user/0/containerd-mount2534958279: apply nydus blob: unpack nydus blob: unpack nydus tar: unpack bootstrap from nydus: parse tar header: rpc error: code = OutOfRange desc = read past object length 54797046 bytes

nydus-image log

time="2023-05-17T15:34:24+08:00" level=debug msg="saved dapdgw7aeqou4bbcymhfxc94n as context:context:" span="[internal] load build context" spanID=7108c9f5c020f1dd traceID=3841af58efa9339c11d3c4107ad1e65a

time="2023-05-17T15:34:24+08:00" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to compute cache key: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown"

failed to compute cache key: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown

20015 0.0.0+unknown buildkitd --config /etc/buildkit/buildkitd.toml

github.com/moby/buildkit/solver.(*edge).createInputRequests.func1.1

/root/buildkit/solver/edge.go:839

github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2

/root/buildkit/solver/internal/pipe/pipe.go:82

runtime.goexit

/root/sdk/go1.18/src/runtime/asm_amd64.s:1571

20015 0.0.0+unknown buildkitd --config /etc/buildkit/buildkitd.toml

main.unaryInterceptor.func1

/root/buildkit/cmd/buildkitd/main.go:576

github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1

/root/buildkit/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25

github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1

/root/buildkit/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34

github.com/moby/buildkit/api/services/control._Control_Solve_Handler

/root/buildkit/api/services/control/control.pb.go:2440

google.golang.org/grpc.(*Server).processUnaryRPC

/root/buildkit/vendor/google.golang.org/grpc/server.go:1340

google.golang.org/grpc.(*Server).handleStream

/root/buildkit/vendor/google.golang.org/grpc/server.go:1713

google.golang.org/grpc.(*Server).serveStreams.func1.2

/root/buildkit/vendor/google.golang.org/grpc/server.go:965

runtime.goexit

/root/sdk/go1.18/src/runtime/asm_amd64.s:1571

使用 nydusaccelerator/buildkit 构建buildkit之后未复现

5.7. dragonflyoss仓库的镜像无法 inspect

nerdctl inspect ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xda8eb3]

goroutine 1 [running]:

github.com/containerd/nerdctl/pkg/inspecttypes/dockercompat.ImageFromNative(0xc0006cc000)

/home/runner/work/nerdctl/nerdctl/pkg/inspecttypes/dockercompat/dockercompat.go:297 +0x253

github.com/containerd/nerdctl/pkg/cmd/image.Inspect.func1({0x13f0190?, _}, {{{0xc0006b0280, 0x3a}, 0x0, {{0xc0006ae0c0, 0x27}, {0xc0006c0050, 0x47}, 0x248, ...}, ...}, ...})

/home/runner/work/nerdctl/nerdctl/pkg/cmd/image/inspect.go:52 +0x205

github.com/containerd/nerdctl/pkg/idutil/imagewalker.(*ImageWalker).Walk(0xc000547568, {0x13f0190, 0xc00014cd20}, {0x7ffc1ceb0732, 0x3a})

/home/runner/work/nerdctl/nerdctl/pkg/idutil/imagewalker/imagewalker.go:81 +0x666

github.com/containerd/nerdctl/pkg/idutil/imagewalker.(*ImageWalker).WalkAll(0x0?, {0x13f0190, 0xc00014cd20}, {0xc0005477c0?, 0x1, 0x103b5e0?}, 0x1)

/home/runner/work/nerdctl/nerdctl/pkg/idutil/imagewalker/imagewalker.go:98 +0xea

github.com/containerd/nerdctl/pkg/cmd/image.Inspect({_, _}, _, {_, _, _}, {{0x13e7d20, 0xc00012e008}, {0x0, 0x0, ...}, ...})

/home/runner/work/nerdctl/nerdctl/pkg/cmd/image/inspect.go:64 +0x1a6

main.inspectAction(0xc000382600, {0xc00034e040, 0x1, 0x1?})

/home/runner/work/nerdctl/nerdctl/cmd/nerdctl/inspect.go:147 +0xb7c

github.com/spf13/cobra.(*Command).execute(0xc000382600, {0xc000120170, 0x1, 0x1})

/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862

github.com/spf13/cobra.(*Command).ExecuteC(0xc000004300)

/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd

github.com/spf13/cobra.(*Command).Execute(...)

/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992

main.xmain()

/home/runner/work/nerdctl/nerdctl/cmd/nerdctl/main.go:136 +0x9f

main.main()

/home/runner/work/nerdctl/nerdctl/cmd/nerdctl/main.go:119 +0x19

原因未知

5.8. nydus镜像与oci镜像manifest的区别

ociv19

{

"schemaVersion": 2,

"mediaType": "application/vnd.oci.image.manifest.v1+json",

"config": {

"mediaType": "application/vnd.oci.image.config.v1+json",

"size": 7023,

"digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"

"layers": [

{

"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",

"size": 32654,

"digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0"

{

"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",

"size": 16724,

"digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"

{

"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",

"size": 73109,

"digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"

}

"subject": {

"mediaType": "application/vnd.oci.image.manifest.v1+json",

"size": 7682,

"digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270"

"annotations": {

"com.example.key1": "value1",

"com.example.key2": "value2"

}

docker v2.2

{

"schemaVersion": 2,

"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",

"manifests": [

{

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

"size": 4710,

"digest": "sha256:016db0f74f4d70593d50a571e76792f1223971e14fb791f9f6fbef3695846295",

"platform": {

"architecture": "amd64",

"os": "linux"

}

{

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

"size": 4708,

"digest": "sha256:1a3cd94d36b750c10782c41bf88458128686cf9b5b81da4bb136dae7c11e439f",

"platform": {

"architecture": "arm",

"os": "linux",

"variant": "v5"

}

{

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

"size": 4708,

"digest": "sha256:fac5d849d932a3a33d421f2918d6b36214fb8f048a04533e95bba5a1da263dc5",

"platform": {

"architecture": "arm",

"os": "linux",

"variant": "v7"

}

{

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

"size": 4709,

"digest": "sha256:6196a365b5ac3c585d65aeb73b68810fc9a24453d5e78ae6a8f229a0d8edf060",

"platform": {

"architecture": "arm64",

"os": "linux",

"variant": "v8"

}

{

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

"size": 4710,

"digest": "sha256:1703758d9c663476da8b6e3ae929a359633d93f217dabb75fc09bda99b46aa84",

"platform": {

"architecture": "386",

"os": "linux"

}

{

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

"size": 4708,

"digest": "sha256:d4089c75201a6f692e55a9b5480ab7bb94a58937d4452d444f06863324d1ccd5",

"platform": {

"architecture": "mips64le",

"os": "linux"

}

{

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

"size": 4710,

"digest": "sha256:bc495de2c53f8e0df08ee58ab06ef2e4fae9304826f649c8285c27a3ee9674eb",

"platform": {

"architecture": "ppc64le",

"os": "linux"

}

{

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

"size": 4708,

"digest": "sha256:bb98ee8ed8c2bf30b82a612775d9218cde446a7ea342d58b1826b2b29aeccaad",

"platform": {

"architecture": "s390x",

"os": "linux"

}

]

}

nydus

{

"schemaVersion": 2,

"mediaType": "application/vnd.oci.image.manifest.v1+json",

"config": {

"mediaType": "application/vnd.oci.image.config.v1+json",

"digest": "sha256:f259d1eb0bf769d60aac8c78e4811a8eeefd811a27c24f6db6c36f72ac5c2324",

"size": 2137

"layers": [

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca",

"size": 32571260,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129",

"size": 295018,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",

"digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab",

"size": 1067099,

"annotations": {

"containerd.io/snapshot/nydus-blob": "true"

}

{

"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",

"digest": "sha256:e26163305101a9bc6d05ad80945a8d360d99a48b0d3ef070cf0c2137655b6115",

"size": 297868,

"annotations": {

"containerd.io/snapshot/nydus-bootstrap": "true",

"containerd.io/snapshot/nydus-fs-version": ""

}

]

}

5.9. 构建的镜像list时告警

nerdctl images

WARN[0000] failed to get unpacked size of image "reg.harbor.com/hello/helloworld:v0.0.7-nydus-nightly-v5" for platform "linux/amd64" error="snapshot sha256:0ee114e6b61fec8478b90a426bdcbbc145d5857d84822922152dc8e09111f391 does not exist: not found"

REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE

reg.harbor.com/hello/helloworld v0.0.7-nydus-nightly-v5 484d07c284ad 33 seconds ago linux/amd64 0.0 B 345.5 MiB

本地镜像存储问题删除本地镜像即可，具体问题尚未可知

5.10. 问题 buildkit打的nydus镜像无法run ,run ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 没问题

现象:

buildkit打的nydus镜像无法run ,run ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 没问题使用nerdctl命令运行也一样

ctr run -d reg.harbor.com/hello/helloworld:v0.0.10-nydus-nightly-v5 hello

错误输出

INFO[0000] apply failure, attempting cleanup error="failed to extract layer sha256:ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown" key="extract-962072102-FeAw sha256:f5294af206581b011d1423a0b7c580b44338d6fb790eaf0ebdcc838fcb107d29"

ctr: failed to extract layer sha256:ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown

构建完使用 nerdctl命令查看有告警信息

nerdctl images

WARN[0000] failed to get unpacked size of image "reg.harbor.com/hello/helloworld:v0.0.10-nydus-nightly-v5" for platform "linux/amd64" error="snapshot sha256:66e2cbc88c92b00741c687d56bedd11654d45c08dec2fcb38406683fb2695135 does not exist: not found"

REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE

xxxx/hello/helloworld v0.0.10-nydus-nightly-v5 b3faf19db1a6 11 minutes ago linux/amd64 0.0 B 32.6 MiB

containerd配置

[proxy_plugins]

[proxy_plugins.nydus]

type = "snapshot"

address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]

snapshotter = "nydus"

disable_snapshot_annotations = false

discard_unpacked_layers = false

环境说明

containerd containerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1buildkitd -v buildkitd github.com/moby/buildkit v0.0.0+unknown 是通过ttps://github.com/nydusaccelerator/buildkit.git 构建的nerdctl -v nerdctl version 1.3.1 在 /etc/nerdctl/nerdctl.toml 指定了 snapshotter debug = false

debug_full = false

address = "unix:///var/run/containerd/containerd.sock"

namespace = "default"

snapshotter = "nydus"

experimental = true

buildkitd 启动命令 NYDUS_BUILDER=/usr/bin/nydus-image buildkitd --oci-worker-snapshotter=nydus --config /etc/buildkit/buildkitd.toml`

构建命令 buildctl build --progress=plain --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=xxxx/helloworld:v0.0.10-nydus-nightly-v5,compression=nydus,oci-mediatypes=true

本地镜像存储问题删除本地镜像即可，问题同上，nydus-snapshotter+fusedev的问题，具体尚未可知，删除本地镜像使用远端镜像即可

5.11. 运行oci-ref镜像导致nydusd进程z状态

使用nydus zran artifact

使用zran artifact镜像启动容器导致nydusd进程d状态

5.11.1. 现象

pod 无法启动，describe po信息

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 33s default-scheduler Successfully assigned default/busybox-nydus-oci-ref to node1

Normal Pulled 31s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 1.035131143s

Normal Pulled 25s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 119.347118ms

Warning Failed 19s (x2 over 25s) kubelet Error: failed to create containerd container: wait until daemon is RUNNING: get daemon state: daemon socket /var/lib/containerd-nydus/socket/cho77etp707gng95ic00/api.sock: not found

Normal Pulling 5s (x3 over 32s) kubelet Pulling image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref"

Normal Pulled 5s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 101.618808ms

查看nydus-snapshotter log

2023-05-26 08:54:19.401685 +00:00] INFO [app/src/lib.rs:74] Program Version: v2.1.5, Git Commit: "ef2033c2e21d36938410069f07d32a5095a19398", Build Time: "2023-03-09T10:09:03.563891101Z", Profile: "release", Rustc Version: "rustc 1.61.0 (fe5b13d68 2022-05-18)"

[2023-05-26 08:54:19.401791 +00:00] INFO [src/bin/nydusd/main.rs:514] Set rlimit-nofile to 1000000, maximum 1048576

[2023-05-26 08:54:19.402224 +00:00] DEBUG [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/pseudo_fs.rs:161] pseudo fs iterate "/"

[2023-05-26 08:54:19.402452 +00:00] INFO [rafs/src/metadata/md_v6.rs:47] rafs superblock features: DIGESTER_SHA256 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESS_GZIP | PRESERVED_INLINED_CHUNK_DIGEST

[2023-05-26 08:54:19.402573 +00:00] ERROR [rafs/src/metadata/layout/v6.rs:1375] RafsV6Blob: idx 0 invalid ci_digest

[2023-05-26 08:54:19.402588 +00:00] ERROR [error/src/error.rs:21] Error:

"invalid Rafs v6 blob entry"

at rafs/src/metadata/layout/v6.rs:1495

note: enable `RUST_BACKTRACE=1` env to display a backtrace

[2023-05-26 08:54:19.402630 +00:00] ERROR [error/src/error.rs:21] Error:

Rafs(FillSuperblock(Os { code: 22, kind: InvalidInput, message: "Invalid argument" }))

at src/bin/nydusd/daemon.rs:140

note: enable `RUST_BACKTRACE=1` env to display a backtrace

[2023-05-26 08:54:19.402669 +00:00] ERROR [src/bin/nydusd/main.rs:664] Failed in starting daemon: Invalid argument (os error 22)

Error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }

time="2023-05-26T08:54:19.404707985Z" level=error msg="Process 14029 has been a zombie"

time="2023-05-26T08:54:19.404759638Z" level=error msg="Nydusd cho77etp707gng95ic00 probably not started"

time="2023-05-26T08:54:19.412716804Z" level=error msg="Process 14029 has been a zombie"

time="2023-05-26T08:54:19.579192096Z" level=error msg="Process 14029 has been a zombie"

time="2023-05-26T08:54:19.719630411Z" level=error msg="Process 14029 has been a zombie"

time="2023-05-26T08:54:19.853129550Z" level=error msg="Process 14029 has been a zombie"

time="2023-05-26T08:54:20.023548928Z" level=error msg="Process 14029 has been a zombie"

time="2023-05-26T08:54:20.181968604Z" level=error msg="Process 14029 has been a zombie"

查看进程进入z状态

[root@node1 nydus]# ps -ef|grep 14029

root 14029 22492 0 08:54 ? 00:00:00 [nydusd]

root 16385 15474 0 08:56 pts/2 00:00:00 grep --color=auto 14029

5.11.2. 解决方式

oci ref 要求 nydus-image/nydusd 2.2

5.12. nydus镜像和oci镜像拉取镜像时间相同

nydus镜像容器创建时最开始只会加载最后一层，所以会很快

{

"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",

"digest": "sha256:ff67b2807cdb0005ac0137bb8d5a4cf453fead83247082f56351fa2ecc3acba0",

"size": 1134668,

"annotations": {

"containerd.io/snapshot/nydus-bootstrap": "true",

"containerd.io/snapshot/nydus-fs-version": "6"

}

对比的时候记得把已经在本地的镜像删除掉

编辑 configmap 增加nydus.toml配置

kubectl edit cm -n nydus-system nydus-snapshotter

apiVersion: v1

data:

config.json: |-

...

nydus.toml: |-

version = 1

address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

daemon_mode = "multiple"

cleanup_on_close = false

[system]

enable = true

address = "/run/containerd-nydus/system.sock"

[system.debug]

daemon_cpu_profile_duration_secs = 5

pprof_address = ""

[daemon]

nydusd_path = "/usr/local/bin/nydusd"

nydusimage_path = "/usr/local/bin/nydus-image"

fs_driver = "fusedev"

recover_policy = "restart"

nydusd_config = "/etc/nydus/config.json"

threads_number = 10

[log]

log_to_stdout = false

level = "debug"

log_rotation_compress = true

log_rotation_local_time = true

log_rotation_max_age = 7

log_rotation_max_backups = 5

log_rotation_max_size = 1

[metrics]

address = ":9110"

[remote]

convert_vpc_registry = false

[remote.mirrors_config]

dir = ""

[remote.auth]

enable_kubeconfig_keychain = false

kubeconfig_path = ""

enable_cri_keychain = false

image_service_address = ""

[snapshot]

enable_nydus_overlayfs = false

sync_remove = false

[cache_manager]

disable = false

gc_period = "24h"

[image]

public_key_file = ""

validate_signature = false

[experimental]

enable_stargz = false

enable_referrer_detect = false

重启nydus-snapshotter

去掉 /etc/containerd/config.toml 中的

[plugins."io.containerd.grpc.v1.cri".containerd]

discard_unpacked_layers = false

disable_snapshot_annotations = false

snapshotter = "nydus"

[proxy_plugins]

[proxy_plugins.nydus]

type = "snapshot"

address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

重启 containerd

systemctl restart containerd

delete nydus-snapshot容器

再次创建即可

5.12.1. nydus镜像启动 nydus-snapshotter日志

nydus镜像会有.rs 输出，这部分是nydus代码用rust编写

time="2023-05-26T11:04:29.478658593Z" level=info msg="[Mounts] snapshot k8s.io/699/987c0c48cdfcbd8b451e7c892d2b922305c3c2fa07e2741ae5d003792dff96f9 ID 457 Kind Active"

time="2023-05-26T11:04:29.838641967Z" level=info msg="[Commit] snapshot with key \"k8s.io/700/extract-821649751-v57_ sha256:17f262ebfabe0a80a19356d4acdde01515c6488b4baf9f474ba805b205be6bcd\" snapshot id 458"

time="2023-05-26T11:04:29.862232365Z" level=info msg="[Commit] snapshot with key \"k8s.io/701/extract-848480697-roQj sha256:4969b12a639d311e01eb2997dca25b3ff4c15297cf1c4332246efc3da416ed90\" snapshot id 459"

time="2023-05-26T11:04:29.884272468Z" level=info msg="[Commit] snapshot with key \"k8s.io/702/extract-875046240-ES_o sha256:eea7439600aea67d552929f639492138db7736625ce46442b556a6773a7e65ad\" snapshot id 460"

time="2023-05-26T11:04:29.902972106Z" level=info msg="[Commit] snapshot with key \"k8s.io/703/extract-893877298-M0p7 sha256:995607f2a156cd5d860cc6cf21d5924430eea1497344c187cef6d8cb5958f08d\" snapshot id 461"

time="2023-05-26T11:04:29.930234997Z" level=info msg="[Commit] snapshot with key \"k8s.io/704/extract-921174867-zDfC sha256:977a5025ef78849159b33940f21f7bc078a4b449a197bf5a84809d18eafa2ecc\" snapshot id 462"

time="2023-05-26T11:04:29.953002744Z" level=info msg="[Commit] snapshot with key \"k8s.io/705/extract-940269515-m4B7 sha256:5008756b9b3af58134566161e811ab255d360cb9f6c03db6be9d1a82b10ea1ce\" snapshot id 463"

time="2023-05-26T11:04:29.985446333Z" level=info msg="[Commit] snapshot with key \"k8s.io/706/extract-963184826-zC2V sha256:dbb29c913fcddebd164aff00387b0cf2ae2605ca7c3ff8ae7fc81e3770c09d6d\" snapshot id 464"

time="2023-05-26T11:04:30.009608437Z" level=info msg="[Commit] snapshot with key \"k8s.io/707/extract-994808807-KS4S sha256:1dc2d75a8c5b4572fcaeb4ee74ee4a34f1f557cf380d85560bea88ddf8e656f1\" snapshot id 465"

time="2023-05-26T11:04:30.034452304Z" level=info msg="[Commit] snapshot with key \"k8s.io/708/extract-19386797-sAab sha256:2e352e3c2a94ff6a53bda70e8c78f1c0061bc5c09918d42f10ee42ca0ea6e05d\" snapshot id 466"

time="2023-05-26T11:04:30.051992401Z" level=info msg="[Commit] snapshot with key \"k8s.io/709/extract-43376118-0fYc sha256:c4655b7ee2454381d27aae6af1bbf99061da67f84943ebf11657db9841d2ce34\" snapshot id 467"

time="2023-05-26T11:04:30.069947216Z" level=info msg="[Commit] snapshot with key \"k8s.io/710/extract-61015950-IVNU sha256:c1775d8affc2abbc45ed4051e72fa8083f624f1a52f938e3dd4017ff2a551216\" snapshot id 468"

time="2023-05-26T11:04:30.088621057Z" level=info msg="[Commit] snapshot with key \"k8s.io/711/extract-79602317-8xBS sha256:7cd39c27735b21390c4f959ca1e12d38c2b73a5f173f5ffd1ee313fee85e317e\" snapshot id 469"

time="2023-05-26T11:04:30.107545892Z" level=info msg="[Commit] snapshot with key \"k8s.io/712/extract-98392114-LQ31 sha256:e835052588cf3a25d66b8a4f683b381ec2314cd293a61d7405861aedce7f1d8f\" snapshot id 470"

time="2023-05-26T11:04:30.138067726Z" level=info msg="[Commit] snapshot with key \"k8s.io/713/extract-129097712-x2We sha256:413163a369a3f699e6971333b345cb9655d22d8850e1713e20766dcc5fc6cf2d\" snapshot id 471"

time="2023-05-26T11:04:30.158431236Z" level=info msg="[Commit] snapshot with key \"k8s.io/714/extract-148574340-w72W sha256:856e93d9260062e3351e85ff977859c11f1c6fcaf4d7de56c50a4a40936ea6dc\" snapshot id 472"

time="2023-05-26T11:04:30.181674205Z" level=info msg="[Commit] snapshot with key \"k8s.io/715/extract-170870922-OaK7 sha256:bfdfb86a11099f8dfdcecb08a46dee87408cb00dedbb69d8c41bd4524e661fb6\" snapshot id 473"

time="2023-05-26T11:04:30.202089436Z" level=info msg="[Commit] snapshot with key \"k8s.io/716/extract-192117639-MjNO sha256:205b438eb857a4428cdfc1a492c3e9e445a7f6aad780f69b082e28471216c9a2\" snapshot id 474"

time="2023-05-26T11:04:30.224765412Z" level=info msg="[Commit] snapshot with key \"k8s.io/717/extract-214032149-U7eo sha256:a72248b4d79a6915f529bb00b97b576fec938895d81c0ebb125a02dd76f91169\" snapshot id 475"

time="2023-05-26T11:04:30.244866341Z" level=info msg="[Commit] snapshot with key \"k8s.io/718/extract-234486897-32MC sha256:f0fc4ec3e74f3762cc5c68435c8c93fb92fcac6ae34053cae00ae90dcafceab6\" snapshot id 476"

time="2023-05-26T11:04:30.428447831Z" level=info msg="[Commit] snapshot with key \"k8s.io/719/extract-255443005-VBbE sha256:3be1111cf1b2f35366acf9d164434690af4860fe8415c7be30faae3de1fde0f3\" snapshot id 477"

time="2023-05-26T11:04:30.495389473Z" level=info msg="Prepares active snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481, nydusd should start afterwards" key=k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 parent="k8s.io/720/sha256:3be1111cf1b2f35366acf9d164434690af4860fe8415c7be30faae3de1fde0f3"

time="2023-05-26T11:04:30.501028169Z" level=info msg="nydusd command: /usr/local/bin/nydusd fuse --config /var/lib/containerd-nydus/config/cho94flp707i10bb5ddg/config.json --bootstrap /var/lib/containerd-nydus/snapshots/477/fs/image/image.boot --mountpoint /var/lib/containerd-nydus/snapshots/477/mnt --apisock /var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock --log-level debug"

[2023-05-26 11:04:30.503482 +00:00] INFO [app/src/lib.rs:74] Program Version: v2.1.5, Git Commit: "ef2033c2e21d36938410069f07d32a5095a19398", Build Time: "2023-03-09T10:09:03.563891101Z", Profile: "release", Rustc Version: "rustc 1.61.0 (fe5b13d68 2022-05-18)"

[2023-05-26 11:04:30.503560 +00:00] INFO [src/bin/nydusd/main.rs:514] Set rlimit-nofile to 1000000, maximum 1048576

[2023-05-26 11:04:30.504138 +00:00] INFO [rafs/src/metadata/md_v6.rs:47] rafs superblock features: DIGESTER_BLAKE3 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESS_ZSTD | PRESERVED_INLINED_CHUNK_DIGEST

[2023-05-26 11:04:30.504414 +00:00] INFO [storage/src/backend/connection.rs:263] backend config: ConnectionConfig { proxy: ProxyConfig { url: "", ping_url: "", fallback: false, check_interval: 5, use_http: false }, mirrors: [], skip_verify: false, timeout: 5, connect_timeout: 5, retry_limit: 2 }

[2023-05-26 11:04:30.631601 +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time: 1685100870

[2023-05-26 11:04:30.647334 +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time: 1685100870

[2023-05-26 11:04:30.668014 +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time: 1685100870

[2023-05-26 11:04:30.918676 +00:00] INFO [src/bin/nydusd/fs_service.rs:206] Rafs imported

[2023-05-26 11:04:30.920073 +00:00] INFO [src/bin/nydusd/fs_service.rs:102] Rafs mounted at /

[2023-05-26 11:04:30.920318 +00:00] INFO [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/transport/fusedev/linux_session.rs:337] mount source rafs dest /var/lib/containerd-nydus/snapshots/477/mnt with fstype fuse opts default_permissions,allow_other,fd=3,rootmode=40000,user_id=0,group_id=0 fd 3

[2023-05-26 11:04:30.920759 +00:00] INFO [src/bin/nydusd/daemon.rs:330] State machine(pid=32141): from Init to Ready, input [Mount], output [None]

[2023-05-26 11:04:30.921167 +00:00] INFO [src/bin/nydusd/daemon.rs:330] State machine(pid=32141): from Ready to Running, input [Start], output [Some(StartService)]

[2023-05-26 11:04:30.921189 +00:00] INFO [src/bin/nydusd/fusedev.rs:318] start 4 fuse servers

[2023-05-26 11:04:30.930559 +00:00] INFO [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:614] FUSE INIT major 7 minor 23

[2023-05-26 11:04:30.930808 +00:00] INFO [src/bin/nydusd/main.rs:660] Fuse daemon started!

[2023-05-26 11:04:30.931000 +00:00] INFO [src/bin/nydusd/api_server_glue.rs:406] HTTP API server running at /var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock

[2023-05-26 11:04:30.931090 +00:00] INFO [api/src/http.rs:911] http server started

[2023-05-26 11:04:30.961941 +00:00] INFO [api/src/http.rs:816] <--- Get Uri { string: "/api/v1/daemon" }

[2023-05-26 11:04:30.962087 +00:00] INFO [api/src/http.rs:821] ---> Get Status Code: OK, Elapse: Ok(146.605µs), Body Size: 866

time="2023-05-26T11:04:30.962507763Z" level=info msg="Nydus remote snapshot 477 is ready"

time="2023-05-26T11:04:30.962543380Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"

time="2023-05-26T11:04:30.964003080Z" level=info msg="Subscribe daemon cho94flp707i10bb5ddg liveness event, path=/var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock."

time="2023-05-26T11:04:30.969335490Z" level=info msg="[Mounts] snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 ID 478 Kind Active"

time="2023-05-26T11:04:30.969369349Z" level=info msg="Nydus remote snapshot 477 is ready"

time="2023-05-26T11:04:30.969416607Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"

time="2023-05-26T11:04:31.124174858Z" level=info msg="[Mounts] snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 ID 478 Kind Active"

time="2023-05-26T11:04:31.124228061Z" level=info msg="Nydus remote snapshot 477 is ready"

time="2023-05-26T11:04:31.124290212Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"

time="2023-05-26T11:04:31.355843306Z" level=info msg="[Mounts] snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 ID 478 Kind Active"

time="2023-05-26T11:04:31.355922313Z" level=info msg="Nydus remote snapshot 477 is ready"

time="2023-05-26T11:04:31.355974771Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"

time="2023-05-26T11:04:31.558021456Z" level=info msg="[Mounts] snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 ID 478 Kind Active"

time="2023-05-26T11:04:31.558049891Z" level=info msg="Nydus remote snapshot 477 is ready"

time="2023-05-26T11:04:31.558096530Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"

5.12.2. oci镜像启动 nydus-snapshotter日志

time="2023-05-26T11:06:30.246293363Z" level=info msg="[Mounts] snapshot k8s.io/722/3d4aa771e42ef99f0d65ce322670115c434297890eaf08bc260eecf2ec3461f7 ID 479 Kind Active"

time="2023-05-26T11:06:33.865401086Z" level=info msg="[Commit] snapshot with key \"k8s.io/723/extract-632530774-VjiP sha256:8cbe4b54fa88d8fc0198ea0cc3a5432aea41573e6a0ee26eca8c79f9fbfa40e3\" snapshot id 480"

time="2023-05-26T11:06:34.704875352Z" level=info msg="[Commit] snapshot with key \"k8s.io/725/extract-249815542-Zc-C sha256:61aba3d78f453ab50b4c0b687156ebb47b6d6dc290f57752d88b737f75a9835c\" snapshot id 481"

time="2023-05-26T11:06:37.941249262Z" level=info msg="[Commit] snapshot with key \"k8s.io/727/extract-722902017-GvJe sha256:d7027bd638e20f68a20d6ae247f2dda259aed1b46a6737bb70de28fa781a1769\" snapshot id 482"

time="2023-05-26T11:06:38.009096474Z" level=info msg="[Commit] snapshot with key \"k8s.io/729/extract-986715165-jE_Q sha256:f116aef2f51af904cebd6baaf86a911036f8540f0837f5081dff235b29135d32\" snapshot id 483"

time="2023-05-26T11:06:38.639504208Z" level=info msg="[Commit] snapshot with key \"k8s.io/731/extract-24233762-aGRw sha256:877496570f634566fc75289c3fd7cc9423f8b42f1b9cf3d5f1ed3703f0ec83e0\" snapshot id 484"

time="2023-05-26T11:06:38.676870045Z" level=info msg="[Commit] snapshot with key \"k8s.io/733/extract-661252421-FKvu sha256:41ca5e8fdea7603d4b22d1bd5dacf18cd8ae4a337c799224034fb38329be6428\" snapshot id 485"

time="2023-05-26T11:06:38.705044906Z" level=info msg="[Commit] snapshot with key \"k8s.io/735/extract-691144330-DHtG sha256:748bab432b39955f9824803e0fb9e2c838c1d82613f6331d4dd06806c6eb08a7\" snapshot id 486"

time="2023-05-26T11:06:38.790391324Z" level=info msg="[Commit] snapshot with key \"k8s.io/737/extract-721825026-kLLd sha256:a7317289186ffc46ccf46108f1a71012bda8de401a9772596b47ac2165279dde\" snapshot id 487"

time="2023-05-26T11:06:38.824921071Z" level=info msg="[Commit] snapshot with key \"k8s.io/739/extract-810354999-OX_U sha256:290e04ebe112a50211c9349710c52fcdba1e90c13c86b7d28fc9d5810fb4ff66\" snapshot id 488"

time="2023-05-26T11:06:39.284365821Z" level=info msg="[Commit] snapshot with key \"k8s.io/741/extract-838355879-06mH sha256:f56f832ba542550bd23ff7bd23bff350112f52c30a25fff73521d22231ae96d2\" snapshot id 489"

time="2023-05-26T11:06:39.318325143Z" level=info msg="[Commit] snapshot with key \"k8s.io/743/extract-302049626-ZzGp sha256:7edd07877746040a573221b32d42a30e71ac050a4b0b52e52637a109f125e07a\" snapshot id 490"

time="2023-05-26T11:06:39.346626770Z" level=info msg="[Commit] snapshot with key \"k8s.io/745/extract-332377616-20YV sha256:8e0edcf4b4ba88591fb18a3189e4ef180a3e907b3d3d374cec1688975c59210a\" snapshot id 491"

time="2023-05-26T11:06:39.373512591Z" level=info msg="[Commit] snapshot with key \"k8s.io/747/extract-359199786-wwaO sha256:0c386a4066a2ba95277419d44500293c0f4fdb017f83fc5f5c8e1047609bc057\" snapshot id 492"

time="2023-05-26T11:06:40.011031985Z" level=info msg="[Commit] snapshot with key \"k8s.io/749/extract-386775080-gUBX sha256:674062350681f3cc06dcc025867235baa315952732ac488d5692347118f5f027\" snapshot id 493"

time="2023-05-26T11:06:40.558428379Z" level=info msg="[Commit] snapshot with key \"k8s.io/751/extract-37315161-xwQT sha256:e723aac1be491e74d1de67af90ddff842b1428f56cc2b36e036c9bb1bf76fd99\" snapshot id 494"

time="2023-05-26T11:06:40.591120152Z" level=info msg="[Commit] snapshot with key \"k8s.io/753/extract-577254979-GuvE sha256:7ac16f932064276888bd1c8be5d3085c99fa945d1545f8b6e4969cc9ba7f932e\" snapshot id 495"

time="2023-05-26T11:06:40.617389492Z" level=info msg="[Commit] snapshot with key \"k8s.io/755/extract-604669307-p-5u sha256:4c8bfefa8feaa457402b6cfbb30bdb73dba02d105a432d4c3cdd3b4c02ba53d1\" snapshot id 496"

time="2023-05-26T11:06:40.650440923Z" level=info msg="[Commit] snapshot with key \"k8s.io/757/extract-630649889-v1Ai sha256:239e5e3542f395b783fe8f614b0299618c2e5fae5acff8d297f81b96bfb65384\" snapshot id 497"

time="2023-05-26T11:06:41.506705121Z" level=info msg="[Commit] snapshot with key \"k8s.io/759/extract-670259326-6llW sha256:bb99dc5df44865572a254bc0ee14ce436b1565097bb0b9010ad90f75dd4e9034\" snapshot id 498"

time="2023-05-26T11:06:41.551350458Z" level=info msg="[Commit] snapshot with key \"k8s.io/761/extract-536500627-OV2m sha256:057bf4d9b4e6fdb652c002e7fd7e34d86832392426ea9a6955fd02b803478958\" snapshot id 499"

time="2023-05-26T11:06:41.617638749Z" level=info msg="[Commit] snapshot with key \"k8s.io/763/extract-602137426-B0nU sha256:92125a743f1a248092f17a7e6e4dcaaa76483e6ff3f1b70ab362889b8c3833ed\" snapshot id 500"

time="2023-05-26T11:06:41.680736991Z" level=info msg="[Mounts] snapshot k8s.io/765/20e960589f1230a38b64c5677fd8eb75a06a0dc704944321b364f78fca4e8f10 ID 501 Kind Active"

time="2023-05-26T11:06:45.304017086Z" level=info msg="[Mounts] snapshot k8s.io/765/20e960589f1230a38b64c5677fd8eb75a06a0dc704944321b364f78fca4e8f10 ID 501 Kind Active"

time="2023-05-26T11:06:45.400978167Z" level=info msg="[Mounts] snapshot k8s.io/765/20e960589f1230a38b64c5677fd8eb75a06a0dc704944321b364f78fca4e8f10 ID 501 Kind Active"

time="2023-05-26T11:06:45.457585857Z" level=info msg="[Mounts] snapshot k8s.io/765/20e960589f1230a38b64c5677fd8eb75a06a0dc704944321b364f78fca4e8f10 ID 501 Kind Active"

6. 参考

参考链接

docker使用nydus启动容器暂不支持buildkit-laily-fetchHow to understand the ‘task’ concept in containerdNerdctl 原生支持 Nydus 加速镜像Nydus | 容器镜像基础Install nydus for containerdhow to check image manifest in containerd?run_nydus_in_kubernetesnydus社区钉钉支持v2list.manifest 主要涉及的github项目

image-servicemoby/buildkitnydusaccelerator/buildkitnydus-snapshotter

nydus-snapshotter ↩︎ buildkitd.toml ↩︎ nydus-design ↩︎ nydus ↩︎ run_nydus_in_kubernetes ↩︎ buildkit-k8s-example ↩︎ How-to-understand-the-‘task’-concept-in-containerd ↩︎ how-to-check-image-manifest-in-containerd? ↩︎ opencontainers/image-spec ↩︎

评论可见，请评论后查看内容，谢谢！！！