
host private ip: whatever ifconfig returns
master_0_ip: 192.168.1.230 # popIP
master_1_ip: 192.168.1.24 # plathIP
master_2_ip: 192.168.1.191 # nerudaIP

# likely will need to change a few files for modem operation or ethernet connection

(old) files that need changing are located in /etc/kubernetes/pki/etcd (the directory for cfssl certs and such) and in /etc/systemd/system/etcd3.service (file that contains service definition)

Sometimes the cluster gets out of sync (in terms of creation and files) and the data directory needs to be nuked on all nodes WITHOUT the etcd3.service running. run (super) rm -r /var/lib/etcd/* and restart with below daemon-reload procedure on all nodes quickly.

----------------------------

Control plane config notes:

Need to stick kube-vip.yaml into /etc/kubernetes/manifests/kube-vip.yaml to set up a virtual ip for the multi-control planes. Note vip value as well, it can be checked after the first control plane is up via kubectl cluster-info. the control plane tweaked join command gets printed when running ./clusterSetup (note changes in clusterConfig.yaml file as well)

----------------------------

Commands to restart etcd service on all master nodes:
sudo systemctl daemon-reload
sudo systemctl enable etcd3.service
sudo systemctl start etcd3.service

----------------------------

etcd.env file located in /etc/etcd.env on each node (each needs the THIS_* variables need to be updated)
# grab new token tbd
TOKEN="token-01"
CLUSTER_STATE=new

NAME_1=pop-os
NAME_2=rossetti
NAME_3=neruda
HOST_1=192.168.1.230
HOST_2=192.168.1.24
HOST_3=192.168.1.191
CLUSTER=pop-os=http://192.168.1.230:2380,rossetti=http://192.168.1.24:2380,neruda=http://192.168.1.191:2380

THIS_NAME=pop-os
THIS_IP=192.168.1.230

----------------------------

etcd3.service file located in /etc/systemd/system/etcd3.service on each node
[Unit]
Description=etcd
Documentation=http://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
EnvironmentFile=/etc/etcd.env
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/usr/bin/etcd \
 --name ${THIS_NAME} \
 --data-dir /var/lib/etcd \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster '${CLUSTER}' \
 --initial-cluster-token ${TOKEN} \
 --initial-cluster-state ${CLUSTER_STATE}

[Install]
WantedBy=multi-user.target


-------------------------------------

setup shit that dont work:

apparently we can run:

docker pull ghcr.io/kube-vip/kube-vip:main

then:

sudo docker run --network host --rm ghcr.io/kube-vip/kube-vip:main manifest pod --interface enp8s0 --vip 192.168.1.240 --arp --leaderElection | sudo tee /etc/kubernetes/manifests/vip.yaml

to get an image that automatically populates the manifests directory for us each time we init w kubeadm... or so they say

--------------------------------

Steps to get load balancer to work, from https://blog.scottlowe.org/2019/08/12/converting-kubernetes-to-ha-control-plane/, we apparently need to start up a cluster and load the kube-vip pod first, THEN move the server address over to the load balancer. upon adding additional control plane nodes we can use the kubeadm config file, but for the first one we maybe need to do this process.

------------------------------

ETCD annoyances. ok it looks like we might need to get fancy  w the certs. There's also some really annoying stuff to do with the original etcd.service being re-made on restart.
how to make the certs and where to put them: https://thenewstack.io/tutorial-set-up-a-secure-and-highly-available-etcd-cluster/
how to make etcdctl work w certs: https://pkg.go.dev/go.etcd.io/etcd/etcdctl#section-readme


NEW new plan, make a local kubernetes cluster basic ass thing. then load up TWO countembaby two etcd clusters, one local to the first control plane and one linked between all control plane's to be

Also something to keep in mind, when we use fancy certs the etcdctl command will non obviously fail (making it look like a networking problem) if the certs are not provided on the cli. easy to gap this one and makes for weird debugging


FROM ETCD:

Disk
An etcd cluster is very sensitive to disk latencies. Since etcd must persist proposals to its log, disk activity from other processes may cause long fsync latencies. The upshot is etcd may miss heartbeats, causing request timeouts and temporary leader loss. An etcd server can sometimes stably run alongside these processes when given a high disk priority.

On Linux, etcd’s disk priority can be configured with ionice:

# best effort, highest priority
$ sudo ionice -c2 -n0 -p `pgrep etcd`


--------------

now getting some sort of error on the load balancer thingy (was using an old and busted version of etcd, updated. using old syntax on kube-vip, updated):

time="2022-03-12T22:54:03Z" level=info msg="server started"
time="2022-03-12T22:54:03Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2022-03-12T22:54:03Z" level=info msg="Namespace [kube-system], Hybrid mode [false]"
time="2022-03-12T22:54:03Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [pop-os]"
I0312 22:54:03.376835       1 leaderelection.go:248] attempting to acquire leader lease kube-system/plndr-cp-lock...
E0312 22:54:03.377316       1 leaderelection.go:330] error retrieving resource lock kube-system/plndr-cp-lock: Get "https://kubernetes:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/plndr-cp-lock": dial tcp 127.0.0.1:6443: connect: connection refused

potential solution: https://forums.rancher.com/t/127-0-0-1-6443-was-refused/36353/2

---> from kubenetes kubeadm documentation (specing apiserver address worked for load balancer):

kubeadm init
It is usually sufficient to run kubeadm init without any flags, but in some cases you might like to override the default behaviour. Here we specify all the flags that can be used to customise the Kubernetes installation.

--apiserver-advertise-address
This is the address the API Server will advertise to other members of the cluster. This is also the address used to construct the suggested kubeadm join line at the end of the init process. If not set (or set to 0.0.0.0) then IP for the default interface will be used.

This address is also added to the certifcate that the API Server uses.

--apiserver-bind-port
The port that the API server will bind on. This defaults to 6443.

--apiserver-cert-extra-sans
Additional hostnames or IP addresses that should be added to the Subject Alternate Name section for the certificate that the API Server will use. If you expose the API Server through a load balancer and public DNS you could specify this with

--apiserver-cert-extra-sans=kubernetes.example.com,kube.example.com,10.100.245.1
--cert-dir
The path where to save and store the certificates. The default is “/etc/kubernetes/pki”.

--config
A kubeadm specific config file. This can be used to specify an extended set of options including passing arbitrary command line flags to the control plane components.


sudo rm -r /var/lib/etcdCluster/
sudo systemctl stop etcd3.service
sudo systemctl daemon-reload
sudo systemctl disable etcd3.service


----------------------------------------------------------------
----------------------------------------------------------------

ok so instead of tearing every last chunk of hair from my skull lets try a stacked etcd topology (the default behaviour of the beast). so now we dont spec any etcd stuff and let them live as pods, unbothered. and that seems to work ok, at least i can see the pods starting up and there's no api server cert issues that I can see. now, for the load balancer there is some weird password/username error.

we generate the manifest with:
sudo docker run --network host --rm ghcr.io/kube-vip/kube-vip:main manifest pod \
--vip 192.168.0.75 \
--arp \
--controlplane \
--leaderElection | sudo tee /etc/kubernetes/manifests/vip.yaml

which both prints the manifest and puts it into the dir for us. still using the kubadm config, we can bring everything up with

sudo kubeadm init -v 9 --config kubeadm-config.yaml --upload-certs

literally forgot the basic home folder set up steps last time so like the dont forget the fundamentals yo. further, the kubeadm configuration file do need to be on each control-plane to be

...

ok so the load balancer manifest likely needs to be put into the remote nodes as well before the join command fires. because i reset the initial node and subsequently kubectl tools stopped working on the secondary control-plane, also there wasn't a kube-vip pod running on the second-added plane.
