This is the first part of a short series. The goal here is narrow and practical: stand up a small but complete Kubernetes cluster — one control plane and two workers — on Fedora CoreOS, built the way you would on-premises. That means a self-managed control plane bootstrapped with kubeadm, your own CNI, and nodes that fully configure themselves on first boot — not a managed service like EKS or GKE where the control plane is someone else’s problem. It runs on DigitalOcean droplets here, but the droplets are standing in for physical servers; the handful of provider-specific touches (importing the OS image, reading each node’s private IP from the metadata service) are called out where they appear.
By the end you have a Butane config that, on first boot, installs containerd and the kubeadm tooling, configures the systemd cgroup driver, and sets every kernel prerequisite kubeadm expects — with no manual SSH step in between — plus the Terraform to provision the nodes and the kubeadm steps to join them into a working cluster.
Two things are deliberately out of scope for this part and will get their own follow-up: Fedora CoreOS’s automatic OS upgrades — and how a self-rebooting node behaves under a live cluster — and the Kubernetes lifecycle, meaning upgrading the cluster across versions. Part 1 stops at a healthy, verified cluster; part 2 puts it through upgrades.
The guide assumes familiarity with regular Linux servers (systemd, sysctl, package managers) and a basic understanding of Kubernetes concepts (control plane, worker nodes, pods). No prior CoreOS experience is required.
How Fedora CoreOS differs
A traditional server OS is mutable. You SSH in, install packages, edit files in /etc, and the machine’s state becomes the sum of every change ever made to it. Fedora CoreOS (FCOS) takes the opposite approach. The OS ships as a single, versioned, immutable image:
- The root filesystem is read-only. You cannot
dnf installonto the running host the usual way. - The whole OS updates as one atomic unit through
rpm-ostree— think of it as “git for your operating system,” with versioned commits and automatic rollback if a new image fails to boot. - Configuration happens once, declaratively, through Ignition, and your workloads run as containers on top of the untouched base.
Because there is no installer and no default password, a fresh image is configured by Ignition: the provisioning system that runs exactly once, on the very first boot. It runs in the initramfs — a small temporary root filesystem the kernel unpacks into memory early in boot, just to bring the system up far enough to find and mount the real on-disk root. Running this early, before the real root filesystem is mounted, is what lets Ignition do things like repartition disks that would be impossible on a running system. You do not write Ignition’s JSON by hand — you write Butane (friendly YAML) and transpile it with the butane tool.
The provisioning pipeline:
flowchart LR
BU["Butane<br/>(YAML)"] -->|"butane transpile"| IGN["Ignition<br/>(JSON)"]
IGN -->|"cloud user-data"| BOOT["First boot<br/>initramfs"]
BOOT -->|"applied once"| HOST["Configured<br/>host"]
style BU fill:#B6D0E2,stroke:#9E9E9E,stroke-width:1px
style IGN fill:#B6D0E2,stroke:#9E9E9E,stroke-width:1px
style BOOT fill:#FFF3E0,stroke:#FF9900,stroke-width:2px
style HOST fill:#E8F5E9,stroke:#388E3C,stroke-width:2pxCreate a Butane file for a Kubernetes node
Start with the login. FCOS has no default password and no console login, so an SSH key for the built-in core user is the only way in. The core user is already a sudoer.
variant: fcos
version: 1.6.0
passwd:
users:
- name: core
ssh_authorized_keys:
# Replace with your own public key
- ssh-ed25519 AAAAC3Nza... you@example.comNext, declare the files that prepare the node. kubeadm needs two kernel modules and three sysctls; overlay backs the container filesystem and br_netfilter lets iptables see bridged traffic, which most CNI plugins require. The config also drops a NetworkManager file that marks Calico’s interfaces (cali*, tunl*, vxlan.calico) as unmanaged. This is essential on FCOS: NetworkManager would otherwise take over those interfaces and break pod routing, leaving every node stuck NotReady — which in turn blocks Calico’s own control-plane pods from scheduling. Setting it up front means the cluster comes up cleanly the first time.
storage:
files:
# No /etc/hostname here on purpose — see "Node identity" below.
# Kernel modules Kubernetes networking needs, loaded on every boot
- path: /etc/modules-load.d/k8s.conf
mode: 0644
contents:
inline: |
overlay
br_netfilter
# Sysctls kubeadm's preflight checks require
- path: /etc/sysctl.d/k8s.conf
mode: 0644
contents:
inline: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
# Stop NetworkManager from managing Calico's interfaces (see note below)
- path: /etc/NetworkManager/conf.d/calico.conf
mode: 0644
contents:
inline: |
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali
# Official Kubernetes 1.33 package repo
- path: /etc/yum.repos.d/kubernetes.repo
mode: 0644
contents:
inline: |
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key
# crictl config so it talks to containerd without --runtime-endpoint flags
- path: /etc/crictl.yaml
mode: 0644
contents:
inline: |
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10Note that there is nothing here to disable swap. kubeadm requires swap to be off, and FCOS does not enable swap by default, so there is nothing to disable.
Layer the packages with systemd
Here the immutable model needs a workaround. You cannot dnf install kubeadm onto a read-only /usr, so a one-shot systemd unit layers the packages with rpm-ostree on first boot and reboots so they take effect. The ConditionPathExists stamp guard ensures it runs only once — Ignition runs once, but this unit lives on disk and would otherwise fire on every boot.
The package set includes cri-tools, which provides crictl — the CLI for inspecting containers and their logs directly through containerd. It is indispensable when a control-plane pod crashes before the Kubernetes API is reachable, since kubectl cannot help you there. The /etc/crictl.yaml written above points it at containerd’s socket, so crictl ps and crictl logs work without passing --runtime-endpoint every time. vim-enhanced is layered too, purely as a convenience — FCOS ships only the minimal vi.
systemd:
units:
# (a) Layer containerd and the Kubernetes tools, then reboot
- name: rpm-ostree-install-k8s.service
enabled: true
contents: |
[Unit]
Description=Layer containerd and Kubernetes packages on first boot
Wants=network-online.target
After=network-online.target
ConditionPathExists=!/var/lib/k8s-layered.stamp
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/rpm-ostree install --idempotent --allow-inactive containerd cri-tools vim-enhanced kubelet kubeadm kubectl
ExecStart=/bin/touch /var/lib/k8s-layered.stamp
ExecStart=/usr/bin/systemctl --no-block reboot
[Install]
WantedBy=multi-user.targetConfigure the containerd cgroup driver
Kubernetes and the container runtime must agree on a cgroup driver. On FCOS (cgroup v2) that means the systemd driver, and a mismatch is the classic “kubelet won’t start” failure. Rather than hand-write a partial config.toml — whose schema differs between containerd 1.x and 2.x — this unit generates containerd’s own default config and flips the one flag, so it is correct for whatever version got layered. It is ordered Before=containerd.service, so the daemon comes up already using the systemd driver.
# (b) Render containerd config with the systemd cgroup driver
- name: containerd-systemd-cgroup.service
enabled: true
contents: |
[Unit]
Description=Render containerd config with the systemd cgroup driver
ConditionPathExists=/usr/bin/containerd
ConditionPathExists=!/var/lib/containerd-config.stamp
Before=containerd.service kubelet.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/mkdir -p /etc/containerd
ExecStart=/bin/sh -c '/usr/bin/containerd config default > /etc/containerd/config.toml'
ExecStart=/usr/bin/sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
ExecStart=/bin/touch /var/lib/containerd-config.stamp
[Install]
WantedBy=multi-user.targetPre-enable the runtime and kubelet by name. Their unit files do not exist yet at provisioning time, so Ignition simply drops the enablement symlinks; they resolve and start once the packages land after the reboot.
# (c) Pre-enable the runtime and kubelet
- name: containerd.service
enabled: true
- name: kubelet.service
enabled: trueFinally, pin the kubelet’s node IP to the private VPC address. On DigitalOcean the kubelet would otherwise register the node under its public IP (the default route). DigitalOcean assigns the VPC address by DHCP, so it cannot be hardcoded — this one-shot unit reads the node’s own private IP from the metadata service and writes it to /etc/sysconfig/kubelet, which kubeadm’s kubelet drop-in already sources. Because it lives in the shared image, it applies to the control plane and every worker identically, which is what lets the worker join later be a single command with no per-node config:
# (d) Register the node under its private VPC IP, read from DO metadata
- name: kubelet-node-ip.service
enabled: true
contents: |
[Unit]
Description=Pin kubelet --node-ip to the private VPC address
Wants=network-online.target
After=network-online.target
Before=kubelet.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'echo KUBELET_EXTRA_ARGS=--node-ip=$$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address) > /etc/sysconfig/kubelet'
[Install]
WantedBy=multi-user.targetAfter a node boots, you can see the result of that autodetection in the file the unit wrote — the kubelet picks up its private VPC address from there on startup:
core@worker-01:~$ cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--node-ip=10.10.10.4The first-boot sequence these units produce:
flowchart TB
A["First boot<br/>Ignition applies config"] --> B["rpm-ostree layers<br/>containerd + kubelet + kubeadm"]
B --> C["Automatic reboot"]
C --> D["containerd starts<br/>systemd cgroup driver"]
D --> E["kubelet starts<br/>node ready for kubeadm"]
style A fill:#FFF3E0,stroke:#FF9900,stroke-width:2px
style B fill:#B6D0E2,stroke:#9E9E9E,stroke-width:1px
style C fill:#A9A9A9,stroke:#9E9E9E,stroke-width:1px
style D fill:#B6D0E2,stroke:#9E9E9E,stroke-width:1px
style E fill:#E8F5E9,stroke:#388E3C,stroke-width:2pxComplete Butane file
variant: fcos
version: 1.6.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-ed25519 AAAAC3Nza... you@example.com
storage:
files:
# Hostname is supplied by the cloud provider (Afterburn), not hardcoded
- path: /etc/modules-load.d/k8s.conf
mode: 0644
contents:
inline: |
overlay
br_netfilter
- path: /etc/sysctl.d/k8s.conf
mode: 0644
contents:
inline: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
- path: /etc/NetworkManager/conf.d/calico.conf
mode: 0644
contents:
inline: |
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali
- path: /etc/yum.repos.d/kubernetes.repo
mode: 0644
contents:
inline: |
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key
- path: /etc/crictl.yaml
mode: 0644
contents:
inline: |
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
systemd:
units:
# Layer containerd and the Kubernetes tools, then reboot
- name: rpm-ostree-install-k8s.service
enabled: true
contents: |
[Unit]
Description=Layer containerd and Kubernetes packages on first boot
Wants=network-online.target
After=network-online.target
ConditionPathExists=!/var/lib/k8s-layered.stamp
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/rpm-ostree install --idempotent --allow-inactive containerd cri-tools vim-enhanced kubelet kubeadm kubectl
ExecStart=/bin/touch /var/lib/k8s-layered.stamp
ExecStart=/usr/bin/systemctl --no-block reboot
[Install]
WantedBy=multi-user.target
# Render containerd config with the systemd cgroup driver
- name: containerd-systemd-cgroup.service
enabled: true
contents: |
[Unit]
Description=Render containerd config with the systemd cgroup driver
ConditionPathExists=/usr/bin/containerd
ConditionPathExists=!/var/lib/containerd-config.stamp
Before=containerd.service kubelet.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/mkdir -p /etc/containerd
ExecStart=/bin/sh -c '/usr/bin/containerd config default > /etc/containerd/config.toml'
ExecStart=/usr/bin/sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
ExecStart=/bin/touch /var/lib/containerd-config.stamp
[Install]
WantedBy=multi-user.target
# Pre-enable the runtime and kubelet
- name: containerd.service
enabled: true
- name: kubelet.service
enabled: true
# Register the node under its private VPC IP, read from DO metadata at boot
- name: kubelet-node-ip.service
enabled: true
contents: |
[Unit]
Description=Pin kubelet --node-ip to the private VPC address
Wants=network-online.target
After=network-online.target
Before=kubelet.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'echo KUBELET_EXTRA_ARGS=--node-ip=$$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address) > /etc/sysconfig/kubelet'
[Install]
WantedBy=multi-user.targetNode identity
It is tempting to write /etc/hostname into the config, but resist it. kubeadm uses the node’s hostname as its Kubernetes node name, which must be unique across the cluster — and if every node boots from this one shared image, every node would claim the same name and collide. Instead, let the platform supply identity: omit the hostname and FCOS (via Afterburn) pulls it from the provider’s instance metadata at boot, so each machine gets its provider-assigned name automatically. The image stays identical; only the identity differs.
On-premises or on bare metal, where there is no metadata service to read, the hostname must be supplied some other way — typically via DHCP or a per-node Ignition config.
Transpile the Butane file to Ignition
Butane is a separate tool and must be installed first. On macOS, install it with Homebrew:
brew install butaneThen transpile the config to Ignition JSON, using --strict to fail on warnings:
butane --pretty --strict k8s-node.bu --output k8s-node.ignDescribe the infrastructure with Terraform
The image, the network, and the nodes could each be created one command at a time with doctl, but every command mutates your account, prints an ID, and leaves you to copy that ID into the next command by hand — which is exactly what the vpc_id/image_id/key_id juggling would otherwise be for. Terraform inverts this: you declare each piece once, and it works out the dependency order, passes IDs between resources by reference, and records what it created in a state file so a second apply is a no-op rather than a duplicate. The three-node loop becomes a single resource with for_each, and there is nothing left to validate by hand — a reference to something that does not exist is a plan-time error.
Install Terraform and give the provider a token through the environment, so no secret lands in the code:
brew install terraform
export DIGITALOCEAN_TOKEN=dop_v1_...Pin the provider so a future init cannot silently jump a major version:
terraform {
required_providers {
digitalocean = {
source = "digitalocean/digitalocean"
version = "~> 2.0"
}
}
}
provider "digitalocean" {}Define a custom DigitalOcean image
DigitalOcean does not ship FCOS as a stock image, so import a current build from the stable stream first. As a digitalocean_custom_image resource, Terraform submits the import and waits until DigitalOcean has processed the qcow2 into a usable image before anything downstream runs:
resource "digitalocean_custom_image" "coreos" {
name = "CoreOS"
url = "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/44.20260523.3.1/x86_64/fedora-coreos-44.20260523.3.1-digitalocean.x86_64.qcow2.gz"
regions = ["fra1"]
}Create a VPC
By default, droplets join DigitalOcean’s shared per-region VPC. For a cluster it is cleaner to give the nodes their own private network, so that intra-cluster traffic — the kubelet, etcd, and the CNI — stays on private IPs, isolated from any other resources in the account. Create it in the same region as the image; every droplet that references it inherits the dependency, so the network is guaranteed to exist before the first node boots:
resource "digitalocean_vpc" "k8s" {
name = "k8s-vpc"
region = "fra1"
ip_range = "10.10.10.0/24"
}Reference the SSH key
The SSH key was uploaded to DigitalOcean once, by hand, and is not something this configuration should own or delete — so it is a data source, a read-only lookup rather than a managed resource. The lookup is by exact name and fails loudly if the key is missing:
data "digitalocean_ssh_key" "default" {
name = "your-ssh-key-name"
}Boot the nodes
Create droplets from that image, handing each the transpiled Ignition file as user-data. The user_data argument is the delivery step from the pipeline diagram — it hands the JSON to the machine, which provisions itself once on first boot. for_each over the set of node names creates all three from one block, and every .id reference wires the image, VPC, and key together without a hand-copied ID:
resource "digitalocean_droplet" "node" {
for_each = toset(["control-01", "worker-01", "worker-02"])
name = "${each.key}.do.dudkin.cz"
image = digitalocean_custom_image.coreos.id
region = "fra1"
size = "s-2vcpu-2gb"
vpc_uuid = digitalocean_vpc.k8s.id
ssh_keys = [data.digitalocean_ssh_key.default.id]
user_data = file("${path.module}/k8s-node.ign")
}Each droplet name (control-01.do.dudkin.cz and so on) becomes that node’s hostname automatically — which is exactly why the Butane config left /etc/hostname out. The same k8s-node.ign provisions all three; their roles are decided later, when you run kubeadm init on control-01 and kubeadm join on the workers.
Expose each node’s public IP as an output, so you do not need a separate list command to find them:
output "node_ips" {
value = { for name, droplet in digitalocean_droplet.node : name => droplet.ipv4_address }
}Then create everything — init downloads the provider, plan previews the image, VPC, and three droplets to add, and apply builds them, waiting until each droplet is active:
terraform init
terraform plan
terraform applyConnect to a node
Fedora CoreOS’s default administrative user is core — there is no password login and no root SSH access. terraform apply prints the node_ips output (reprint it any time with terraform output node_ips); pick an address and connect as core:
terraform output node_ips
ssh core@<droplet-public-ip>You may have noticed the SSH key is specified twice: once in the Butane config under passwd, and again with ssh_keys on the droplet. These are two independent mechanisms that happen to overlap on DigitalOcean. The key in Ignition is written straight into core’s authorized_keys and works on any platform. The ssh_keys argument instead registers the key in DigitalOcean’s instance metadata. Keeping both is harmless, and setting ssh_keys also stops DigitalOcean from falling back to emailing a root password.
Complete Terraform files
provider.tf:
terraform {
required_providers {
digitalocean = {
source = "digitalocean/digitalocean"
version = "~> 2.0"
}
}
}
provider "digitalocean" {}droplets.tf:
resource "digitalocean_custom_image" "coreos" {
name = "CoreOS"
url = "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/44.20260523.3.1/x86_64/fedora-coreos-44.20260523.3.1-digitalocean.x86_64.qcow2.gz"
regions = ["fra1"]
}
resource "digitalocean_vpc" "k8s" {
name = "k8s-vpc"
region = "fra1"
ip_range = "10.10.10.0/24"
}
data "digitalocean_ssh_key" "default" {
name = "your-ssh-key-name"
}
resource "digitalocean_droplet" "node" {
for_each = toset(["control-01", "worker-01", "worker-02"])
name = "${each.key}.do.dudkin.cz"
image = digitalocean_custom_image.coreos.id
region = "fra1"
size = "s-2vcpu-2gb"
vpc_uuid = digitalocean_vpc.k8s.id
ssh_keys = [data.digitalocean_ssh_key.default.id]
user_data = file("${path.module}/k8s-node.ign")
}
output "node_ips" {
value = { for name, droplet in digitalocean_droplet.node : name => droplet.ipv4_address }
}Bootstrap the cluster
The Butane config gets each node fully prepared, but it deliberately stops short of forming a cluster. kubeadm init creates a control plane, so running it on every node would produce three one-node clusters instead of one. It also generates a join token and CA certificate hash at runtime — values that do not exist until the cluster is created and therefore cannot be baked into a static image. Bootstrapping is inherently a post-boot step, which is also why the same image can serve every node.
There is one Fedora CoreOS-specific trap to disarm first. Flexvolume is Kubernetes’ older storage-driver interface: volume plugins ship as executables dropped into a directory on each node, which the kubelet and kube-controller-manager scan to attach external storage to pods. It predates today’s CSI standard but is still wired into kubeadm’s defaults. By default that plugin directory is /usr/libexec/kubernetes/kubelet-plugins/volume/exec/, and kubeadm mounts the path into the kube-controller-manager static pod with “create if missing”. But on FCOS /usr is read-only, so the directory cannot be created, the controller-manager pod never starts, and kubeadm init hangs and fails with kube-controller-manager check failed ... connection refused. The apiserver and scheduler come up fine, which makes the failure look mysterious.
The fix is to point that directory at a writable path. We use /var/lib/kubelet/volumeplugins — a writable location under kubelet’s own state directory — and we will give Calico the exact same path later, so the whole cluster is consistent.
Pass a small kubeadm config file instead of command-line flags. It does three things: relocates the flexvolume directory, carries the pod network CIDR (Calico’s default 192.168.0.0/16), and — just as importantly on DigitalOcean — pins the control plane to the node’s private VPC address. Left alone, kubeadm advertises the public IP because that is the default route, which is exactly how the API server and the etcd server/peer certificates end up bound to an internet-facing address.
There is a catch, though: you cannot hardcode that address, because DigitalOcean assigns the VPC IP by DHCP when the droplet is created — you do not know it ahead of time. Read it from the droplet’s metadata service instead and generate the config from it. SSH to core@control-01 and run:
# control-01's private VPC IP from DigitalOcean's metadata service
PRIV_IP=$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: ${PRIV_IP}
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
networking:
podSubnet: 192.168.0.0/16
controllerManager:
extraArgs:
# Default is /usr/libexec/... which is read-only on Fedora CoreOS
- name: flex-volume-plugin-dir
value: /var/lib/kubelet/volumeplugins
EOF
sudo kubeadm init --config kubeadm-config.yamlWhen it finishes, kubeadm prints a kubeadm join ... command containing the token and CA cert hash. Copy it — the workers need it verbatim. Then set up kubectl for the core user, as the init output instructs:
mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configInstall Calico
Until a CNI plugin is installed, every node stays NotReady and pods cannot get IP addresses. Install Calico with the Tigera operator. The cluster was initialized with Calico’s default CIDR (192.168.0.0/16), so the IP pool in the stock custom-resources-bpf.yaml already matches:
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/v1_crd_projectcalico_org.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/tigera-operator.yaml
curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/custom-resources-bpf.yaml
kubectl create -f custom-resources-bpf.yamlThis is where read-only /usr bites a second time. Calico’s calico-node DaemonSet installs a flexvolume driver under /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ by default — and just like kube-controller-manager earlier, kubelet cannot create that directory on FCOS. The calico-node pods get stuck initializing with mkdir /usr/libexec/kubernetes: read-only file system, so the nodes never reach Ready. Point the operator at the same writable path we gave kube-controller-manager and it re-renders the DaemonSet automatically:
kubectl patch installation default --type=merge \
-p '{"spec":{"flexVolumePath":"/var/lib/kubelet/volumeplugins"}}'One more VPC detail: Calico autodetects each node’s IP with its first-found method, which on DigitalOcean can latch onto the public interface. Pin it to the VPC range so pod and BGP traffic stay private too:
kubectl patch installation default --type=merge \
-p '{"spec":{"calicoNetwork":{"nodeAddressAutodetectionV4":{"cidrs":["10.10.10.0/24"]}}}}'Watch the Calico pods finish initializing; once calico-node is running on every node, they all report Ready:
watch kubectl get pods -n calico-systemJoin the workers
Joining a worker is now a one-liner, because the kubelet already knows to register under its private IP — the kubelet-node-ip.service unit in the Butane config set that on every node at boot. On control-01, print a ready-to-run join command (it already points at the control plane’s private endpoint, since that is what the API server advertises):
kubeadm token create --print-join-commandCopy the output, SSH to each worker, and run it with sudo:
ssh core@<worker-01-ip>
sudo kubeadm join 10.10.10.3:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>Repeat on worker-02. That is the whole join — no per-worker config file, and nothing hardcoded, because the private node IP is handled by the node itself.
Verify the cluster
Back on control-01, all three nodes should report Ready once Calico has programmed their networking:
core@control-01:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
control-01.do.dudkin.cz Ready control-plane 6m33s v1.33.13 10.10.10.3 <none> Fedora CoreOS 44.20260523.3.1 7.0.9-205.fc44.x86_64 containerd://2.2.3
worker-01.do.dudkin.cz Ready <none> 5m50s v1.33.13 10.10.10.4 <none> Fedora CoreOS 44.20260523.3.1 7.0.9-205.fc44.x86_64 containerd://2.2.3
worker-02.do.dudkin.cz Ready <none> 93s v1.33.13 10.10.10.2 <none> Fedora CoreOS 44.20260523.3.1 7.0.9-205.fc44.x86_64 containerd://2.2.3The cluster is up: one control plane and two workers, all on the private VPC, with Calico handling pod networking.
graph TB
subgraph VPC["DigitalOcean VPC (10.10.10.0/24)"]
CP["control-01<br/>control plane"]
W1["worker-01"]
W2["worker-02"]
end
W1 -->|"kubeadm join :6443"| CP
W2 -->|"kubeadm join :6443"| CP
style VPC fill:#F3E5F5,stroke:#7B1FA2,stroke-width:2px
style CP fill:#E8F5E9,stroke:#388E3C,stroke-width:2px
style W1 fill:#B6D0E2,stroke:#9E9E9E,stroke-width:1px
style W2 fill:#B6D0E2,stroke:#9E9E9E,stroke-width:1pxProduction notes
This guide builds a working cluster, but two things about it are deliberately kept simple and must not be carried into production unchanged.
First, security. The kubeadm config above already pins the API server and etcd to the private VPC address, so they do not advertise on the internet-facing interface — but every DigitalOcean droplet still has a public IP attached, and nothing blocks inbound traffic to it by default. The Kubernetes API (6443), etcd (2379-2380), and the kubelet (10250) must never be reachable from the open internet: anyone who can reach 6443 can hammer it, and an exposed etcd is a full cluster takeover. Put a firewall in front of every node that allows 6443 only from trusted addresses and blocks etcd and the kubelet ports entirely, and keep all inter-node traffic on the private VPC interface (which, with the private advertiseAddress and node-ip set above, is already where it flows).
Second, availability. This cluster has a single control-plane node, which makes it a single point of failure. If control-01 goes down you lose the API server and, with stacked etcd, the cluster’s only copy of its state — the workers keep running their pods, but you can no longer schedule, scale, or recover anything. A production control plane runs an odd number of nodes (three or five) behind a load balancer, so etcd keeps quorum and the API stays available when one node is lost. With Fedora CoreOS that matters even more, since nodes reboot themselves to apply updates: a single control plane will have unavoidable downtime windows, whereas an HA control plane rides through them. Treat the single-node control plane here as a learning and staging shape, not a production one. (How Fedora CoreOS’s self-reboots for updates interact with a running cluster, and how to upgrade Kubernetes itself, are the subject of part 2.)