project on hold for now due to "out of capacity" issues with the huge arm instances.
This repo contains a complete deployment of a working k8s cluster on the oracle cloud just using free tier components. Everything is done via IAC so no manual deployment necessary.
non exhaustive list of features:
- vms and network infra deployment via OpenTofu
- k8s-Cluster Deployment via Ansible
- hostnames and dns-names for applications via external-dns using cloudflare
- letsencryt certificates via traefik
- ingress controller is traefik
- networking cni is cilium
- secrets-management via external-secrets on oci secrets manager
- deployment of applications and k8s-components via argocd
oci = oracle cloud infrastructure
Networking is changed now to public subnet and all instances as public instances only. I did not create a new diagram so the image above is not the actual implemented infrastructure. linux distri with a very current kernel necessary, maybe fedora.
https://gist.github.com/olegstepura/e18098bf1367f86f2bb2b7dbbc49b6f9
delete all dns entries: use the script provided in cloudflare https://gist.github.com/slayer/442fa2fffed57f8409e0b23bd0673a92
domain: virtual-lab.org
hostnames/dns entries: control-node1. control-node2. worker1. worker2. argocd.
using k8s the hard way as a tutorial and then create ansible playbooks out of it
- cilium as cni and to replace kube-proxy (very current kernel necessary): https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/
- external-secrets using oci secrets manager
- external-dns using cloudflare
- traefik as ingress controller and as a letsencryt cert-solver via http01: https://medium.com/kubernetes-tutorials/deploying-traefik-as-ingress-controller-for-your-kubernetes-cluster-b03a0672ae0c
credits:
Actually the idea for this specific infrastructure came from here: https://medium.com/geekculture/how-to-create-an-always-free-k8s-cluster-in-oracle-cloud-60be3b107c44 and of course the great kubernetes-the-hard-way repo: https://github.com/kelseyhightower/kubernetes-the-hard-way
https://library.tf/providers/oracle/oci/latest/docs/resources/core_cpe
export TF_VAR_private_key=`openssl rsa -in private.key -check`
tofu init
tofu plan
tofu applySeems to be like a container for aa whole infrastructure inside my oci account https://docs.oracle.com/en-us/iaas/developer-tutorials/tutorials/tf-compartment/01-summary.htm#
https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/accesscontrol.htm#Policies https://docs.oracle.com/en-us/iaas/Content/Identity/Concepts/commonpolicies.htm
actually its a bit more complicated:
to make a loadbalancer policy run a tenancy is needed. "Allow group ${oci_identity_group.k8s-cluster-on-oci_admins.name} to manage load-balancers in tenancy"
for user, groug and policy tenancy is necessary compartment_id = var.tenancy_ocid
vault type must be vault_type = "DEFAULT"
https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm#compute
2 VM.Standard.E2.1.Micro 4 OCPU and 24GB VM.Standard.A1.Flex
the arm instances are not available so it makes no sense..
https://docs.oracle.com/en-us/iaas/images/
For now I will use: Oracle-Linux-9.3-aarch64-2024.03.25-0 has kernel 5.15
Maybe I will switch to ubuntu: Canonical-Ubuntu-22.04-aarch64-2024.02.18-0 has also kernel 5.15 but might be upgradable to 6.2! When it is published I will use: Canonical-Ubuntu-24.04
200 GB of Block Volume. five volume backups. 50 GB boot volume for storage per instance 20 GB of Always Free Overview of Object Storage.
All master encryption keys protected by software are free. All tenancies get 20 key versions of master encryption keys protected by a hardware security module (HSM) and 150 Always Free Vault secrets.
2 Oracle Autonomous Databases. 1 Oracle NoSQL Database with up to 133 million reads per month, 133 million writes per month, and 3 tables with 25 GB storage per table.
1 Flexible Load Balancer Min and Max 10 Mbps. 1 Network Load Balancer. 2 VCN.
Free OCI's Bastion service provides restricted and time-limited Secure Shell Protocol (SSH) access to target resources that don't have public endpoints.
{
"compartment-id": "ocid1.tenancy.oc1..aaaaaaaav75hekijieedwtzyf3e3kvelgbodmdwn3gyj44doyhhezpsdkira",
"id": "ocid1.availabilitydomain.oc1..aaaaaaaaiifj24st3w4j7cowuo3pmqcuqwjapjv435vtjmgh5j7q3flguwna",
"name": "wXwm:EU-FRANKFURT-1-AD-1"
},
{
"compartment-id": "ocid1.tenancy.oc1..aaaaaaaav75hekijieedwtzyf3e3kvelgbodmdwn3gyj44doyhhezpsdkira",
"id": "ocid1.availabilitydomain.oc1..aaaaaaaaa2artt5wizbqvwl3rgptylx2l7jqbnyv4dygcfvlrd3dphvi3mdq",
"name": "wXwm:EU-FRANKFURT-1-AD-2"
},
{
"compartment-id": "ocid1.tenancy.oc1..aaaaaaaav75hekijieedwtzyf3e3kvelgbodmdwn3gyj44doyhhezpsdkira",
"id": "ocid1.availabilitydomain.oc1..aaaaaaaalcdcbl7u6akbmkojxhrozpj2v7yavqqydkj3ytyjbt47lnoqnm2q",
"name": "wXwm:EU-FRANKFURT-1-AD-3"
}https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/compute-clusters.htm
seems like the free arm instances are very rare. so terraform is throwing “Out of Capacity” issue. Well Project is on Hold for now. As this does not make sense.
https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/overviewIPsec.htm#top
https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingpublicIPs.htm
NAT-Gateway and Public Load Balancer always get Public IPs. Load-Balancer: regional reserved public IP => IP can be chosen or regional ephemeral public IP => Not changeable NAT-Gateway: regional ephemeral public IP => Not changeable
https://docs.oracle.com/en-us/iaas/Content/GSG/Tasks/loadbalancing.htm
Empty Security List Route Table with
Target Type: Select Internet Gateway. Target: Select your VCN's internet gateway. Destination CIDR Block: Enter 0.0.0.0/0.
https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/NATgateway.htm
Route Table with:
Target Type: NAT Gateway. Target NAT Gateway: The NAT gateway. Destination CIDR Block: 0.0.0.0/0
https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/dns.htm