This question drove me to rebuild my home lab Kubernetes cluster. On the original build, I loved to tinker around with Kubernetes objects, deploy a test app here, another one there. However this is not what is considered a scalable solution to run workloads on. Time to power up the cluster!
My hardware setup:
Odd setup, right? The Eeros are what I am using to supply wifi for my house. They are cheap for the features and specs that they provide. Eero also pioneered home wifi mesh networks. They don’t provide many ethernet ports to plug into, and I wanted a wired network for the Raspberry Pi’s to avoid packet failure. Insert Juniper EX2200C switch here. Now the raspberry pi’s are connected into a wired network to avoid as much interference has possible, and I can access the Raspberry Pi’s through my home wifi!
Now it was time to look into the Operating System to install on each of the Raspberry Pi’s. I wanted something that was easy to set up and security was top priority. Also, if the Operating System comes with Kubernetes installed out of the box, that is a BIG win. One Operating System jumped to the top, and it was one that I was familiar with. Enter Talos Linux.
Talos Linux markets themselves as the “Kubernetes Operating System”. The Operating System is minimal, and locked down. The only way to make changes to the system is by interfacing with it through an API using mutual TLS encryption. This felt like Kubernetes, which I am on board with. So I wiped the SD cards and burned the Talos Linux image onto them.
Lesson Learned: When bringing up the Kubernetes side of the install, I wanted to use Calico CNI instead of the default Flannel. The configs for Talos Linux included a commented line to configure Calico instead, so I uncommented it and tried to bring up the Kubernetes cluster. Something was wrong with that specific version, and some of the pods did not come up. Changing the version that Calico was using immediately fixed the issue. There were a few documented issues within Github against that version. Although I could not find a correlation between my issue and the ones authoring the issue, something seemed to be linked and a newer version came up without a problem.
Now we have a Kubernetes cluster up. What’s next? I felt like these questions or tasks needed to be addressed that would help streamline any developer’s experience:
More to come…
]]>I have seen that statement in a few places. Although I don’t agree with the statement, I understand why people are saying it. Kubernetes provides an easy model to deploy and schedule resources for applications. Also, there are loads of tools to make it easier to deploy apps the way you would like to deploy them.
I love managing everything, including my apps, through git. You can easily look at your git repository and determine what is deployed and how it is configured. This methodology has been named Git Ops, and there are a few tools that help with detecting changes to git and keeping your Kubernetes cluster in sync. One tool that provides this functionality is Argo CD.
Argo CD is very flexible, defining app deployments through yaml. At the surface, I have heard of a concept with Argo CD named “App of Apps”. Argo CD has the power to manage applications and keep it in sync with git. Argo CD can even manage itself. This functionality sounds promising, and I knew I had to dig deep to uncover it’s secrets.
I found this blog by Arthur Koziel to help me understand Argo CD with Helm. His blog was to the point and covered the topic at the appropriate level needed to start getting exposure to Argo CD, Helm, and the concept of “App of Apps”.
Even though Arthur does a great job breaking things down, I want to summarize a little here for my own understanding.
First trick to getting started with Argo CD is deploying it. The easiest way is to deploy it using Helm. Helm is a templating tool for Kubernetes manifests. Arthur recommended running these commands after putting together some basic yaml for configuration of Argo CD:
helm repo add argo-cd https://argoproj.github.io/argo-helm
helm dep update charts/argo-cd/
The first command adds the repository containing Argo’s Charts (the company, not the tool) to your Helm app so it can interact with charts related to Argo CD. The second command downloads 2 components:
After setting all of these files up, we can then use Helm to install Argo CD by running:
helm install argo-cd charts/argo-cd/
Now we can start configuring the root app that will help to track apps deploying to the k8s cluster. After configuring a chart for the root app, we apply it with this:
helm template apps/ | kubectl apply -f -
To install new apps, we add a template to the chart using the argoproject API configuration in the yaml, and viola! The new app will appear in the cluster after an update to the repository.
If we are managing Argo CD through Argo CD, we will need to tell Helm that it doesn’t need to manage Argo CD as an app anymore. We can run this command:
kubectl delete secret -l owner=helm,name=argo-cd
This was a great introduction to Argo CD and Helm. Next things to look into are:
All of my code for this blog is located here. This repository is bound to change after posting this blog post.
]]>Before I get into my Python learnings, I have to describe the problem I am trying to address.
My current project can be summed up as building automation around a brown field network environment with hundreds of network devices and varying standards through years of implementation. Also, the older implmentations were never brought into compliance. All of the network devices have different Operating Systems, configurations and standards that I have to build an automation pipeline for. Sound familiar? This is my challenge, it has been an interesting challenge.
I looked at network automation frameworks and came across 2 leading tools: (Ansible)[https://www.ansible.com] and (Nornir)[https://nornir.readthedocs.io/en/latest/]. I will not go into great detail on the comparisons between the 2 and why we went with Nornir (might come in another post). I learned a little Python before coming to the NetOps team. I loved that I could use the tooling I already know how to use with Python! Also, I saw limitations with the growth of the project and being limited to yaml files. I know Ansible is extensible and has a huge community behind it. Coming from Python, it felt limiting.
I got my team sold on Nornir. Nornir has a great inventory system, manages my connnections to the network devices, and can be extensible through a plugin system (introduced in 3.0). Also, it uses common, open source projects to define tasks to be run against network devices. Already coming from Python, it made the most sense to me to use Nornir because it was just another Python Library.
While building out the automation project with Nornir, I have looked at many Python projects to see how they are organized, and what are some common practices and coding styles between each project. It has helped me learn Python and code project management. I will be documenting it in future parts. Please look forward to future posts!
]]>My next goal is to become more proficient with administering a k8s cluster. So I started searching for recommendations and guides to install k8s on a set of Raspberry Pi’s (I had 2 version 4 Raspberry Pi’s). Microk8s kept rising to the top of the list with many installation guides to use. I stuck with Canonical’s Guide, since they made Microk8s and Canonical has a decent track record with making and maintaining Ubuntu.
I read through the instructions first and this looked easy and straight forward. Install Ubuntu 20.04 like you would install Raspbian on Rapberry Pi. Then use snap to install Microk8s and start it up. Well, there were a few gotchas.
At the time of this writing, 1.19 stable version would always fail with ‘127.0.0.1:16443 connection refused’. Snap would rollback the install with nothing more than that response. I found a github issue against Microk8s expressing this same issue. Great! Install the edge release and move on.
Next issue I faced was trying to get Microk8s to start. Well, I found another issue, but this was a LOT easier to find. Running microk8s inspect showed me that I had to enable cgroups. Doing this changed has now allowed me to play with Microk8s.
I can add nodes into the cluster, and start having fun with Microk8s.
]]>I started doing research on static website creation because I think that WordPress is heavy for a simple blog site. I haven’t felt any need to have people comment against my posts. If someone wants to reach me, I link a few contact options. The added benefit to move away from WordPress is getting less spam comments!
Jekyll was the first static website creator I looked into. I decided to learn Jekyll because I stumbled across GitHub Pages. Github Pages is free, making it an easy decision to play with GitHub Pages. I found that it utilizes a git repository with Jekyll. Also, Jekyll is built on Ruby, which I am familiar with. Bottom line, it seemed like a really good match for me.
Running through the Getting Started Guide, I found that it has great flexibility with adding pages, and templating, making it easier to build a quick site for documentation. If I need something complex, I can write a quick Ruby script. Also, I can use markdown and html files. This provides me with great flexibility for how I want display my information.
The final thing that I love about static website creation was how easy it is to integrate it with git, which I use everyday at work. This provides great control over my site through easy revision history.
I want to spend time in the future to explore Sphinx because I work with Python regularly. Hugo also interested me, since Go interests me.
]]>I just started studying for the AWS Certified Developer Associate Certification. I used A Cloud Guru’s material when studying for my AWS Solutions Architect Associate Certification. Because of the success I had using their material, I am using them for the Developer Associate Certification. One of the topics covered is Elastic Load Balancing (ELB) and the different types of ELB’s that AWS offers.
Today, there are 3 different types of ELB’s: Classic, Application, and Network. Classic is the original ELB that the Elastic Compute Cloud (EC2), operating at the connection and request levels. Application Load Balancer (ALB) is marketed to be best suited for HTTP/HTTPS traffic. Network ELB’s are best suited for high performance traffic needs operating at the Layer 4 of the OSI Model. When learning about the 3 types of ELB’s, I asked “when would I use the Classic ELB over the ALB?”
Classic ELB’s are documented to still be used for any instances built as EC2 Classics, meaning not built in an AWS Virtual Private Cloud (VPC). I VPC has lots of benefits and is the recommended to build out new EC2 infrastructure in the AWS Cloud. If the ELB is not load balancing between EC2 Classic Instances, AWS recommends usage of the ALB. Is there any other reasons to use Classic ELB’s over ALB’s?
Cost is always a driving factor. ELB’s and ALB’s have different cost structures. Looking at the pricing page for ELB’s, Classic ELB’s are priced based on run time per hour and the amount of data in GB processed by the ELB. ALB’s are priced based on run time per hour and Load Balancer Capacity Units (LCU). Looking further into their documentation, LCU’s are a unit of measurement based on 4 dimensions: new connections, active connections, bandwidth, and rule evaluations. AWS will then charge based on the higher used dimension.
What does this mean? Know your traffic, and do your homework. Analyze the traffic that is supposed to hit your ELB and work out the cost based on the criteria of the Classic ELB and the ALB. You might find that the Classic ELB is cheaper.
ELB’s are very powerful, providing the ability to scale your application resiliency to traffic load variation or high traffic throughput. Even though ELB’s are powerful, they will add to your overall infrastructure costs. Do your homework so you know what the up front cost will be before the monthly AWS bill hits your email.
]]>The first thing I found that I needed from my hosting service was an affordable option for site encryption. My former hosting service didn’t provide one. I loved what I was seeing with LetsEncrypt. It constantly rotates the certificate, and web browsers accepted certificates from LetsEncrypt. Also, it’s free!
Unfortunately, my former hosting service did not allow me to have root privileges, which I needed to have in order to install the LetsEncrypt agent. Exhausting all other options to get encryption for my blog site at a reasonable cost, I decided it was time to shop around.
I found a lot of hosting providers provide great deals for new customers, cutting my hosting costs by almost half. Also, some of the hosting providers provided affordable options for encryption. They would allow me to set up encryption with LetsEncrypt for free. Or if I needed a higher level of trust with my certificate, they would provide one for an additional cost. Since this is my personal blog site, free is great!
Now that I have determined a hosting provider that cuts my hosting costs, has great reviews from customers, and does site encryption for free, I paid my hosting fee. Now time to start exporting my blog from one host to another.
Exporting the data from my former hosting service was not that painful. I quickly found my inexperience with the software hosting providers use for multi-tenanted hosting when I was trying to import my data. I spent about 3 hours one day trying to import my MySQL backup into the portal of my new hosting provider with no luck.
I finally stumbled across a blog post that gave me the information I was looking for. It is a simple point and click to get the backup into MySQL. I realized that I was trying to use my operations experience to do this work. I tried to get access to the database through logging into the host, and uploading the backup into the database. It just did not work. Hosting providers want common and easy tooling for their users. PHP and phpMyAdmin are common and easy for web developers. Figuring out about the single point and click option in the portal was a game changer.
At the command line, it kept saying I didn’t have root privileges, even though the credentials I created were supposed to be root. I was trying to use my systems experience, which is not applicable for blog hosting providers. For a number of reasons, it held me up from bringing my site up for nearly a year. Frustration and life got the best of me.
What I took away from this experience is sometimes you need to think outside of the box. Also, relate the tool to the common user that is expected to use it. With my experiences coming from working in operations, that is how I approached my problem of loading in a backup of my MySQL database. But the hosting provider has to deal with a different set of users. The customer wants GUI’s. They want things “point and click”. They want it easy.
]]>I am utilizing the Raspbian Operating System on my Raspberry Pi. It is the recommended OS for the Raspberry Pi, and it is based on Debian Linux. I am very familiar with Linux growing up outside of the world of Windows. One crucial package provides functionality for mDNS.
Multicast DNS is a great way to set up communication on a network with minimal setup. MacOS and Raspbian support mDNS out of the box. Windows requires an additional package. The easiest way to get mDNS on Windows is to install iTunes. Have you heard of Bonjour? It is included with iTunes. It is also using mDNS.
mDNS communicates on a known IP Address that is broadcasted on the local network. Anyone who is talking mDNS is also listening on that same address. This is done automatically, sharing hostnames and IP Addresses so others on the network know how to reach each other. I can plug my MacBook directly up to the Raspberry Pi via a patch cable. After both devices default to Link Local Addressing, I ping “raspberrypi.local” from my MacBook and success!
Pinging a device and seeing success feels great, but you will want to connect to it. Make sure to add a file named “ssh” to the root of the MicroSD Card after cloning the Raspbian image to the card.
mDNS simplifies initial connection to the Raspberry Pi. This has helped me to easily build automation to deploy projects with a base image of Raspbian.
References:
https://en.wikipedia.org/wiki/Multicast_DNS
https://raspberrypi.stackexchange.com/questions/58478/ssh-not-working-with-fresh-install
]]>