Documentation

Storage & registries

Persistent volumes, CSI drivers, the cluster database, and where images come from.

Local volumes (local-path)

KubeSolo deploys Rancher's local-path-provisioner (v0.0.36) by default. It creates a local-path StorageClass, marked as the cluster default, that provisions PersistentVolumes as directories on the node.

PropertyValue
StorageClasslocal-path (default class)
Provisionerrancher.io/local-path
Volume bindingWaitForFirstConsumer
Reclaim policyRetain: data stays on disk after the PVC is deleted
Volume directory/var/lib/kubesolo/local-path-storage
Namespacelocal-path-storage
pvc.yaml
1apiVersion: v1
2kind: PersistentVolumeClaim
3metadata:
4 name: data
5spec:
6 accessModes: [ReadWriteOnce]
7 resources:
8 requests:
9 storage: 1Gi

Because the reclaim policy is Retain, deleting a claim leaves its directory and PersistentVolume behind. Delete the PV and its directory yourself when the data is no longer needed.

Turning it off

Local storage is on by default. To turn it off, set storage.localPath.enabled: false and restart, or pass --local-storage=false to the install script. That only stops KubeSolo deploying it on later starts. If it is already running, also remove it with kubectl delete namespace local-path-storage and kubectl delete storageclass local-path.

Volumes on a shared filesystem

storage.localPath.sharedPath makes the provisioner create volumes under a path you choose, such as an NFS or other shared mount, using local-path's shared filesystem mode instead of the data directory:

yaml
1storage:
2 localPath:
3 sharedPath: /mnt/shared

At install time, set KUBESOLO_LOCAL_STORAGE_SHARED_PATH with sudo -E. Verify the provisioner configuration with kubectl -n local-path-storage get cm local-path-config -o yaml.

CSI drivers and the kubelet path

Full CSI drivers work on KubeSolo. One detail matters: the kubelet's root directory is under the data directory, not the upstream default.

kubelet root directory
1/var/lib/kubesolo/kubelet

Operators, add-ons and CSI drivers that assume /var/lib/kubelet need their kubelet directory setting pointed at KubeSolo's path. For example, the NFS CSI Helm chart exposes kubeletDir:

bash
1$ helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
2$ helm repo update
3$ helm upgrade --install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --set kubeletDir=/var/lib/kubesolo/kubelet

If you change path at install time, the kubelet directory moves with it: <path>/kubelet.

The cluster database

Cluster state lives in SQLite through Kine, which serves the etcd API to the API server on 127.0.0.1:2379. The database file is /var/lib/kubesolo/kine/state.db.

Two settings help on edge hardware:

SettingDefaultUse it when
storage.dbWALRepairfalseDevices lose power without a clean shutdown. At startup KubeSolo checks the database and removes the WAL and SHM files (state.db-wal, state.db-shm) if it finds corruption.
kubernetes.apiServer.startupTimeoutSeconds600Storage is slow, such as SD cards, and components take longer than ten minutes to pass their startup health checks.

Registry mirrors and private registries

KubeSolo's embedded containerd reads per-registry configuration from hosts.toml files, using containerd's standard hosts directory format. No flags are involved; create files in this directory:

layout
1/var/lib/kubesolo/containerd/registry/
2└── <registry-host>/
3 └── hosts.toml

containerd reads the files at pull time, so you do not need to restart after creating or changing them. Use fully qualified image references, such as docker.io/library/nginx:alpine, so each pull maps to the right directory.

Mirror for Docker Hub

/var/lib/kubesolo/containerd/registry/docker.io/hosts.toml
1server = "https://registry-1.docker.io"
 
3[host."https://mirror.corp.internal"]
4 capabilities = ["pull", "resolve"]

If the mirror does not have an image, containerd falls back to registry-1.docker.io.

Harbor proxy cache, private CA and credentials

Harbor's API is scoped by project, so the mirror URL carries the project path and override_path = true stops containerd adding its own /v2/:

/var/lib/kubesolo/containerd/registry/docker.io/hosts.toml
1server = "https://registry-1.docker.io"
 
3[host."https://harbor.corp.internal/v2/docker.io"]
4 capabilities = ["pull", "resolve"]
5 override_path = true
6 ca = "/etc/ssl/certs/harbor-ca.crt"
7 [host."https://harbor.corp.internal/v2/docker.io".header]
8 Authorization = ["Basic <base64 of robot$account:token>"]

Create one directory per upstream registry (docker.io, ghcr.io, quay.io and so on).

Air-gapped: one registry for everything

The special _default directory catches every registry that has no directory of its own. Point it at an internal registry that holds your workload images:

/var/lib/kubesolo/containerd/registry/_default/hosts.toml
1[host."https://airgap-registry.corp.internal"]
2 capabilities = ["pull", "resolve"]
3 ca = "/etc/ssl/certs/corp-ca.crt"

A registry-specific hosts.toml always takes precedence over _default. Combine this with the offline build, which embeds KubeSolo's own system images.

Local registry without TLS

/var/lib/kubesolo/containerd/registry/localhost:5000/hosts.toml
1server = "http://localhost:5000"
 
3[host."http://localhost:5000"]
4 capabilities = ["pull", "resolve", "push"]
5 skip_verify = true

Verifying

bash
1$ kubectl run registry-test --image=docker.io/library/nginx:alpine --restart=Never
2$ kubectl delete pod registry-test

Use an image that is not already cached on the node, then check the mirror's logs for the pull. Registry configuration applies to the embedded containerd only. With an external runtime, configure registries in that runtime.