All-in-3 Openshift (OCP) cluster with OCS (storage) and CNV (Virtualization)

Another day, another “Edge” Architecture. This time let’s see how the minimum all-in-one OCP/OCS/CNV would have to look like.

But first what are the key benefits:

  • Small footprint
  • High Availability
  • Virtualization with VM HA
  • HCI storage
  • Best containers platform there is

Okay, I got you this far, let me reward you now with a short 2 minutes demo on how this is going to work once you’re done:

I have to warn you, this is not a fully approved/tested setup and you would likely need a support exception if you wanted Red Hat to support you on this design.

Also don’t one in all-in-one be misleading. We really need to start with 4 hosts (1x bootstrap and 3x masters/workers). By the end of this example we get to run it all on just 3 nodes.

Red Hat official documentation:


You need to start with pre-defining your hardware. This is a minimum spec that worked for me:

1x Bootstrap20G61x pxe 1x baremetal100GB
3x Master/worker52GB241x pxe 1x baremetal150GB +250GB

Bootstrap Configuration

  1. Log into Bootstrap machine (I configured my based on registered RHEL8 with dedicated kni user)

ssh kni@<ip-learned-from-horizon>  

  1. Updating system

[kni@bootstrap ~]$ sudo yum update -y

Query to see if bootstrap node needs restarted:

[kni@bootstrap ~]$ needs-restarting -r Core libraries or services have been updated since boot-up:   * dbus   * dbus-daemon   * kernel   * linux-firmware   Reboot is required to fully utilize these updates. More information:   ** needs-restarting is part of the yum-utils package

[kni@bootstrap ~]$ sudo reboot  (if needed ^)

  1. Installing the KNI Packages

[kni@bootstrap ~]$ sudo dnf install -y libvirt qemu-kvm mkisofs python3-devel jq ipmitool

  1. Modify the user to add the libvirt group to the newly created user.

[kni@bootstrap ~]$ sudo usermod –append –groups libvirt kni

  1. Start and enable libvirtd

[kni@bootstrap ~]$ sudo systemctl enable libvirtd –now

  1. Create the default storage pool and start it.

[kni@bootstrap ~]$ sudo virsh pool-define-as –name default –type dir –target /var/lib/libvirt/images

Pool default defined

[kni@bootstrap ~]$ sudo virsh pool-start default

Pool default started

[kni@bootstrap ~]$ sudo virsh pool-autostart default

Pool default marked as autostarted

  1. Set up networking

   export PROV_CONN=”eth1″

   sudo nmcli con down “$PROV_CONN”

   sudo nmcli con delete “$PROV_CONN”

   sudo nmcli con down “System $PROV_CONN”

   sudo nmcli con delete “System $PROV_CONN”

   sudo nmcli connection add ifname provisioning type bridge con-name provisioning

   sudo nmcli con add type bridge-slave ifname “$PROV_CONN” master provisioning

   sudo nmcli connection modify provisioning ipv4.addresses ipv4.method manual

   sudo nmcli con down provisioning

   sudo nmcli con up provisioning

   export BM_CONN=”eth2″

   sudo nmcli con down “$BM_CONN”

   sudo nmcli con delete “$BM_CONN”

   sudo nmcli con down “System $BM_CONN”

   sudo nmcli con delete “System $BM_CONN”

   sudo nmcli connection add ifname baremetal type bridge con-name baremetal

   sudo nmcli con add type bridge-slave ifname “$BM_CONN” master baremetal

   sudo nmcli connection modify baremetal ipv4.addresses ipv4.method manual

   sudo nmcli con down baremetal

   sudo nmcli con up baremetal

  1. Verify the networking looks similar to this:

[kni@bootstrap ~]$ nmcli con show

NAME               UUID                                  TYPE      DEVICE       

System eth0        5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03  ethernet  eth0         

baremetal          7a452607-2196-49d2-a145-6cd760a526eb  bridge    baremetal    

provisioning       6089a4a6-d3d3-4fff-8d99-04ec920d2754  bridge    provisioning 

bridge-slave-eth1  40e4e145-a948-4832-b51a-9373ef5f0d45  ethernet  eth1         

bridge-slave-eth2  7f73af05-d3e7-4564-8212-9a66a8f1f347  ethernet  eth2  

  1. Create a pull-secret.txt file.

[kni@bootstrap ~]$ vi pull-secret.txt

In a web browser, navigate to Install on Bare Metal with user-provisioned infrastructure, and scroll down to the Downloads section. Click Copy pull secret. Paste the contents into the pull-secret.txt file and save the contents in the kni user’s home directory.

Openshift Installation

  1. Retrieving OpenShift Installer GA

[kni@bootstrap ~]$ export VERSION=latest-4.7

[kni@bootstrap ~]$ export RELEASE_IMAGE=$(curl -s$VERSION/release.txt | grep ‘Pull From:’ | awk -F ‘ ‘ ‘{print $3}’)

[kni@bootstrap ~]$ export cmd=openshift-baremetal-install

[kni@bootstrap ~]$ export pullsecret_file=~/pull-secret.txt

[kni@bootstrap ~]$ export extract_dir=$(pwd)

[kni@bootstrap ~]$ curl -s$VERSION/openshift-client-linux.tar.gz | tar zxvf – oc


[kni@bootstrap ~]$ sudo cp oc /usr/local/bin

[kni@bootstrap ~]$ oc adm release extract –registry-config “${pullsecret_file}” –command=$cmd –to “${extract_dir}” ${RELEASE_IMAGE}

  1. Verify the installer file has been downloaded

[kni@bootstrap ~]$ ls

GoodieBag  nohup.out  oc  openshift-baremetal-install  pull-secret.txt

  1. Create install-config.yaml.

4. Here is a good template on how your all-in-3 install-config shold look like:

[kni@bootstrap ~]$ vi GoodieBag/install-config.yaml

apiVersion: v1

basedomain: hexo.lab


  name: “kni-aio”



  networkType: OVNKubernetes


– name: worker

  replicas: 0


  name: master

  replicas: 3


    baremetal: {}






    provisioningNetworkInterface: ens3


      – name: “kni-worker1”

        bootMACAddress: “fa:16:3e:50:10:bf”

        #role: worker


          deviceName: “/dev/vda”

        hardwareProfile: default


          address: “ipmi://X.X.X.X”

          username: “kni-aio”

          password: “Passw0rd”

      – name: “kni-worker2”

        bootMACAddress: “fa:16:3e:46:fd:6c”

        #role: worker


          deviceName: “/dev/vda”

        hardwareProfile: default


          address: “ipmi://X.X.X.X”

          username: “kni-aio”

          password: “Passw0rd”

      – name: “kni-worker3”

        bootMACAddress: “fa:16:3e:4c:6f:18”

        #role: worker


          deviceName: “/dev/vda”

        hardwareProfile: default


          address: “ipmi://X.X.X.X”

          username: “kni-aio”

          password: “Passw0rd”

pullSecret: ‘<pull_secret>’

sshKey: ‘<ssh_pub_key>’

5. You pull secret should be available in home directory

(kni-test) [kni@bootstrap ~]$ cat pull-secret.txt 


6. The ssh key should be in your default directory, but feel free to generate on if it’s not there

[kni@bootstrap ~]$ ssh-keygen 

[kni@bootstrap ~]$ cat .ssh/ 

7. Openshift Baremetal IPI also requires you to provide DHCP and DNS. This is what I have used in my lab for that”

[kni@bootstrap ~]$ cat GoodieBag/hosts   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 bootstrap.kni-aio.hexo.lab bootstrap kni-worker1.kni-aio.hexo.lab kni-worker1 kni-worker2.kni-aio.hexo.lab kni-worker2 kni-worker3.kni-aio.hexo.lab kni-worker3  api.kni-aio.hexo.lab api ns1.kni-aio.hexo.lab ns1

[kni@bootstrap ~]$ cat GoodieBag/kni.dns

#Wildcard for apps — make changes to cluster-name (openshift) and domain (

#Static IPs for Masters

[kni@bootstrap ~]$ cat GoodieBag/resolv.conf.upstream
search hexo.lab
nameserver <my_public_dns>

[kni@bootstrap ~]$ cat GoodieBag/resolv.conf
search hexo.lab
nameserver <my_public_dns>

[kni@bootstrap ~]$ sudo cp GoodieBag/hosts /etc/

[kni@bootstrap ~]$ sudo cp GoodieBag/kni.dns /etc/dnsmasq.d/

[kni@bootstrap ~]$ sudo cp GoodieBag/resolv.conf.upstream /etc/

[kni@bootstrap ~]$ sudo cp GoodieBag/resolv.conf /etc/

[kni@bootstrap ~]$ sudo systemctl enable –now dnsmasq

Created symlink /etc/systemd/system/ → /usr/lib/systemd/system/dnsmasq.service.

8. Finally we should be ready to deploy Openshift cluster

Cleanup first if this is not your first attempt

for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {‘print $2’});


  sudo virsh destroy $i;

  sudo virsh undefine $i;

  sudo virsh vol-delete $i –pool default;

  sudo virsh vol-delete $i.ign –pool default;


[kni@bootstrap ~]$ rm -rf ~/clusterconfigs/auth ~/clusterconfigs/terraform* ~/clusterconfigs/tls ~/clusterconfigs/metadata.json

  1. Create working directory

[kni@bootstrap ~]$ mkdir ~/clusterconfigs

  1. Deploy (in tmux)

[kni@bootstrap ~]$ cp GoodieBag/install-config.yaml clusterconfigs/

[kni@bootstrap ~]$ ./openshift-baremetal-install –dir ~/clusterconfigs –log-level debug create cluster

  1. You can monitor your deployment in another window

[kni@bootstrap ~]$ sudo virsh list

 Id   Name                        State


 1    kni-test2-ntpgv-bootstrap   running

[kni@bootstrap ~]$ sudo console <bootstrap-vm>

Some other commands that will come in handy in the later state of the deployment

[kni@bootstrap ~]$ export KUBECONFIG=/home/kni/clusterconfigs/auth/kubeconfig

[kni@bootstrap ~]$ oc get clusteroperators

[kni@bootstrap ~]$ oc get nodes

(kni-aio) [kni@bootstrap ~]$ oc get nodes

NAME          STATUS   ROLES           AGE   VERSION

kni-worker1   Ready    master,worker   49m   v1.19.0+2f3101c

kni-worker2   Ready    master,worker   48m   v1.19.0+2f3101c

kni-worker3   Ready    master,worker   48m   v1.19.0+2f3101c

  1.  I hope it works! If it has, then at the end of the deployment you will see something like this:

INFO Install complete!                            

INFO To access the cluster as the system:admin user when using ‘oc’, run ‘export KUBECONFIG=/home/kni/clusterconfigs/auth/kubeconfig’ 

INFO Access the OpenShift web-console here: https://console-openshift-console.apps.kni-aio.hexo.lab 

INFO Login to the console with user: “kubeadmin”, and password: “ss4nv-kmhMP-dM6c7-edSVb” 

DEBUG Time elapsed per stage:                      

DEBUG     Infrastructure: 50m41s                   

DEBUG Bootstrap Complete: 12m39s                   

DEBUG  Bootstrap Destroy: 13s                      

DEBUG  Cluster Operators: 38m12s                   

INFO Time elapsed: 1h42m24s   

Deploying OCS (storage)

Official docs ->

  1. Install local storage operator
  1. Keep the defaults and just hit install
  1. I have preconfigured worker nodes  with 250GB secondary drives

[kni@bootstrap ~]$ ssh core@kni-worker1

Red Hat Enterprise Linux CoreOS 46.82.202010301241-0

  Part of OpenShift 4.6, RHCOS is a Kubernetes native operating system

  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,

make configuration changes via `machineconfig` objects:

[core@kni-worker1 ~]$ sudo fdisk -l | grep vdb

Disk /dev/vdb: 250 GiB, 107374182400 bytes, 209715200 sectors

  1. Create local disk

[kni@bootstrap ~]$ cat localstorage.yaml 
apiVersion: “”
kind: “LocalVolume”
  name: “local-disks”
  namespace: “openshift-local-storage” 
    – matchExpressions:
        – key:
          operator: In
          – kni-worker1
          – kni-worker2
          – kni-worker3
    – storageClassName: “localblock-sc”
      volumeMode: Block  
        – /dev/vdb

[kni@bootstrap ~]$ oc create -f localstorage.yaml  

  1. Verify local storage has been create

[kni@bootstrap ~]$ oc get all -n openshift-local-storage

NAME                                         READY   STATUS    RESTARTS   AGE

pod/diskmaker-discovery-jpcxt                1/1     Running   0          6m6s

pod/diskmaker-discovery-lv7m7                1/1     Running   0          6m6s

pod/diskmaker-discovery-x66cs                1/1     Running   0          6m6s

pod/local-disks-local-diskmaker-ccmsx        1/1     Running   0          91s

pod/local-disks-local-diskmaker-gqxm9        1/1     Running   0          92s

pod/local-disks-local-diskmaker-sz9p7        1/1     Running   0          91s

pod/local-disks-local-provisioner-bkt2b      1/1     Running   0          92s

pod/local-disks-local-provisioner-lw7t7      1/1     Running   0          92s

pod/local-disks-local-provisioner-rjt2c      1/1     Running   0          92s

pod/local-storage-operator-fdbb85956-2bb85   1/1     Running   0          10m

NAME                                     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE

service/local-storage-operator-metrics   ClusterIP   <none>        8383/TCP,8686/TCP   9m45s

NAME                                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE

daemonset.apps/diskmaker-discovery             3         3         3       3            3           <none>          6m6s

daemonset.apps/local-disks-local-diskmaker     3         3         3       3            3           <none>          92s

daemonset.apps/local-disks-local-provisioner   3         3         3       3            3           <none>          92s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/local-storage-operator   1/1     1            1           10m

NAME                                               DESIRED   CURRENT   READY   AGE

replicaset.apps/local-storage-operator-fdbb85956   1         1         1       10m

[kni@bootstrap ~]$ oc get pv


local-pv-109d302d   250Gi      RWO            Delete           Available           localblock-sc            3m45s

local-pv-b5506d82   250Gi      RWO            Delete           Available           localblock-sc            3m45s

local-pv-dd42bba8   250Gi      RWO            Delete           Available           localblock-sc            3m45s

  1. Install ‘OpenShift Container Storage’ operator (leave all defaults)
  1. Create OCS cluster

For the All-in-one use case we need to label the nodes so they are available to OCS

for NODE in {1..3};do

oc label nodes kni-worker$NODE”


[kni@bootstrap ~]$ cat localstorage.yaml 
apiVersion: “”
kind: “LocalVolume”
  name: “local-disks”
  namespace: “openshift-local-storage” 
    – matchExpressions:
        – key:
          operator: In
          – kni-worker1
          – kni-worker2
          – kni-worker3
    – storageClassName: “localblock-sc”
      volumeMode: Block  
        – /dev/vdb
[kni@bootstrap ~]$ cat ocs-storage.yaml 
kind: StorageCluster
  name: ocs-storagecluster
  namespace: openshift-storage
  manageNodes: false
  monDataDirHostPath: /var/lib/rook
  – count: 1
        – ReadWriteOnce
            storage: 250Gi
        storageClassName: localblock-sc
        volumeMode: Block
    name: ocs-deviceset
    placement: {}
    portable: false
    replica: 3
    resources: {}

[kni@bootstrap ~]$ oc create -f ocs-storage.yaml    

Configuring container registry on OCS ->

Configuring CNV ->


It might feel like a lot of steps, but most of the work has been presented in here, so you’re welcome 🙂

At the end you will enjoy really powerful 3 node cluster that can host not just your legacy apps in VMs, but also set you up on the road to running in containers.

Leave a Reply

Your email address will not be published. Required fields are marked *