Overview
In the previous post, we set up a homelab Kubernetes cluster using Dell OptiPlex Micro machines and completed the basic configuration. This post covers installing ArgoCD, a GitOps tool for declaratively managing all cluster components from a Git repository, and applying the App of Apps pattern to build a scalable infrastructure management system.
Understanding GitOps
What is GitOps?
GitOps is an operational model first proposed by Alexis Richardson of Weaveworks in 2017. It uses a Git repository as the Single Source of Truth for infrastructure and application configurations. All infrastructure changes are tracked through Git commits, reviewed via Pull Requests, and reflected in the actual environment through automated processes, enabling infrastructure to be managed like code.
Traditional infrastructure management often meant administrators logging directly into servers and making changes by hand, either with shell commands or through management consoles. That made it difficult to track change history, slowed down root cause analysis and recovery when mistakes caused failures, and made it harder to keep multiple environments consistent. GitOps addresses these issues by defining infrastructure as code in Git repositories and having automated tools continuously compare the Git state with the actual cluster state, synchronizing when differences appear.
Core Principles of GitOps
The GitOps methodology is based on four core principles:
- Declarative: Define the desired state of the system declaratively rather than imperatively, storing it in a Git repository in the form of “this is what it should be.” Kubernetes YAML manifests are a prime example.
- Versioned: All changes are recorded as Git commits, allowing tracking of who changed what, when, and why. When problems occur, you can immediately restore to a previous state by rolling back to a specific commit.
- Automatically Applied: Approved changes are automatically applied to the system without manual intervention, preventing human error and increasing deployment speed.
- Continuously Reconciled: Software agents continuously compare the desired state defined in the Git repository with the actual system state, automatically adjusting when differences occur to prevent drift.
Benefits of GitOps
Adopting the GitOps approach provides the following advantages:
- Audit Trail: All infrastructure changes are recorded in Git history, which is useful for compliance audits and failure root cause analysis.
- Enhanced Collaboration: The code review process through Pull Requests can also be applied to infrastructure changes, enabling knowledge sharing and quality improvement among team members.
- Easier Disaster Recovery: Since the entire infrastructure configuration is stored as code in the Git repository, the same state can be quickly reconstructed in a new environment during cluster failures.
- Environment Consistency: Managing development, staging, and production environment configurations from the same codebase minimizes problems caused by environment differences.
Introduction to ArgoCD
What is ArgoCD?
ArgoCD is a declarative continuous deployment tool for Kubernetes built around GitOps. It was developed by Intuit and released as open source in 2018, and later became a CNCF (Cloud Native Computing Foundation) graduated project. It automatically synchronizes Kubernetes manifests stored in Git repositories to clusters and provides a web UI and CLI for monitoring application status.
ArgoCD uses a pull-based deployment model. Unlike the push model, where external CI systems directly access clusters for deployment, ArgoCD runs inside the cluster and continuously polls Git repositories to detect and apply changes. This model offers higher security by avoiding external exposure of cluster credentials and makes it easier to deploy to clusters behind network firewalls.
Core Components of ArgoCD
ArgoCD consists of several components, each performing the following roles:
- API Server: The central component that handles all requests through web UI, CLI, and gRPC/REST API, and manages authentication and authorization.
- Repository Server: Responsible for fetching manifests from Git repositories and running template tools like Helm, Kustomize, and Jsonnet to generate final Kubernetes resources.
- Application Controller: The core controller that continuously compares the desired state defined in Git repositories with the actual cluster state, performing synchronization when differences occur.
- Dex: An OpenID Connect (OIDC) provider that supports SSO (Single Sign-On) integration, enabling connection with external authentication systems like GitHub, GitLab, and LDAP.
- Redis: An in-memory data store used for application state caching and session management.
Core Concepts in ArgoCD
There are two core concepts to understand when using ArgoCD:
- Application: The basic unit of ArgoCD that defines a group of Kubernetes resources. It connects a source (Git repository path) with a destination (Kubernetes cluster and namespace) to specify which manifests to deploy where.
- Project: A policy container that logically groups multiple Applications and restricts access permissions, allowed source repositories, and deployable clusters and namespaces. It is used for resource isolation and security in multi-tenant environments.
Installing ArgoCD
ArgoCD can be installed in several ways, but in my homelab I used Helm because it was the simplest way to keep the installation under GitOps control.
What is Helm?
Helm is a package manager for Kubernetes applications. It was first developed by Deis (now Microsoft) in 2015 and is currently maintained as a CNCF graduated project. It defines complex Kubernetes applications in a package format called “Charts,” can apply different settings per environment through templates and values files, and fills a role similar to apt on Linux or Homebrew on macOS for Kubernetes.
Installing Helm
I started by installing Helm:
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
This command downloads and runs the official Helm install script. After that, I checked the installed version with:
helm version
version.BuildInfo{Version:"v3.12.0", GitCommit:"...", GitTreeState:"clean", GoVersion:"go1.20.4"}
Creating the ArgoCD Namespace
I created a dedicated namespace for ArgoCD:
kubectl create namespace argocd
namespace/argocd created
Installing the ArgoCD Helm Chart
Then I added the official ArgoCD Helm chart repository:
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
And installed ArgoCD with:
helm upgrade --install argocd argo/argo-cd --namespace argocd
The upgrade --install option is idempotent: it upgrades ArgoCD if it is already installed, and installs it if it is not. That makes the command safe to run repeatedly. When the installation finished, it printed the following message:
Release "argocd" does not exist. Installing it now.
NAME: argocd
LAST DEPLOYED: Tue Feb 25 12:34:56 2025
NAMESPACE: argocd
STATUS: deployed
REVISION: 1
Verifying Installation
After installation, I checked the Pod status first:
kubectl get pods -n argocd
NAME READY STATUS RESTARTS AGE
argocd-application-controller-5f8c95f7b8-5xglw 1/1 Running 0 5m
argocd-dex-server-7589cfcbb9-ntzwx 1/1 Running 0 5m
argocd-redis-74cb89f446-c6jsb 1/1 Running 0 5m
argocd-repo-server-6dddb4b65d-gx9vh 1/1 Running 0 5m
argocd-server-54f988d66b-l69zc 1/1 Running 0 5m
If all Pods are in Running status and the READY column shows 1/1, ArgoCD has been successfully installed.
Retrieving the Initial Admin Password
The initial admin password for the ArgoCD web UI is stored in a Kubernetes secret, so I retrieved it with:
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
This command extracts and decodes the base64-encoded password from the secret. I saved the generated password immediately and changed it after logging in.
Accessing the Web UI
For the initial access, I simply used port forwarding:
kubectl port-forward svc/argocd-server -n argocd 8080:443
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Access https://localhost:8080 in a web browser to see the ArgoCD login screen. ArgoCD uses a self-signed certificate by default, so the browser may display a security warning. In development environments, you can safely ignore the warning and proceed.
Enter the username admin and the initial password retrieved earlier to log in and view the ArgoCD dashboard.
Designing the GitOps Repository Structure
To make ArgoCD manageable in this homelab, I split responsibilities across two Git repositories:
- app-of-apps repository: Repository defining the top-level bootstrap application, managing the list of applications to deploy to the cluster and their settings.
- k8s-resource repository: Repository containing actual Kubernetes resources and Helm charts, managing the specific configuration of each application.
This split reduces management complexity by keeping bootstrap logic separate from the actual resource definitions. It is also useful from a security perspective because each repository can have different access permissions.
App of Apps Pattern
What is the App of Apps Pattern?
The App of Apps pattern is a design pattern for hierarchically managing multiple applications in ArgoCD. It has a structure where one root Application creates and manages multiple child Applications. With this pattern, adding a new application is usually just a matter of adding another directory to the Git repository. It also gives you a single entry point for understanding the overall cluster configuration.
In my setup, the App of Apps pattern works roughly like this:
- Root Application Creation: The administrator applies the root Application manifest to the cluster.
- Child Application Creation: The root Application references the Git repository and automatically creates child Applications.
- Actual Resource Deployment: Each child Application deploys the manifests from its referenced Git path to the cluster.
app-of-apps Repository Structure
The first repository looks like this:
app-of-apps/
├── Chart.yaml
├── templates/
│ └── infra-apps-root.yaml
└── values.yaml
This repository follows the Helm chart format. The infra-apps-root.yaml file in the templates/ directory defines an ArgoCD Application that references the ApplicationSet in the second repository.
k8s-resource Repository Structure
The second repository looks like this:
k8s-resource/
├── applicationset.yaml
└── apps/
├── example-app/
│ ├── Chart.yaml
│ ├── templates/
│ └── values.yaml
└── another-app/
├── Chart.yaml
├── templates/
└── values.yaml
In this structure, each subdirectory under the apps/ directory represents one application, and the ApplicationSet automatically detects these directories and creates ArgoCD Applications.
Configuring ArgoCD Applications
Creating the Root Application
I saved the following manifest as app-of-apps.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: app-of-apps
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/injunweb/app-of-apps.git
targetRevision: HEAD
path: .
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
This Application uses the root directory of the app-of-apps repository as its source. The syncPolicy.automated setting automatically detects changes in the Git repository and applies them to the cluster. prune: true automatically deletes resources from the cluster that were deleted from the Git repository, and selfHeal: true automatically restores resources manually changed in the cluster to the Git repository state.
Then I applied it to the cluster:
kubectl apply -f app-of-apps.yaml
application.argoproj.io/app-of-apps created
Configuring the infra-apps-root Application
The templates/infra-apps-root.yaml file in the app-of-apps repository was set up like this:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: infra-apps-root
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/injunweb/k8s-resource.git
targetRevision: HEAD
path: .
directory:
recurse: false
include: "applicationset.yaml"
destination:
server: {{ .Values.spec.destination.server }}
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
This Application fetches and applies only the applicationset.yaml file from the root directory of the k8s-resource repository. The directory.include setting allows selective inclusion of specific files.
Configuring the ApplicationSet
What is ApplicationSet?
ApplicationSet is an ArgoCD feature that uses templates and Generators to automatically create and manage multiple Applications. It can dynamically create Applications based on Git repository directory structures, cluster lists, external data sources, and more. This makes it useful for larger multi-cluster environments or repositories with many services, where a single definition can create and maintain many Applications.
The applicationset.yaml file in the k8s-resource repository looks like this:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: infra-apps
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/injunweb/k8s-resource.git
revision: HEAD
directories:
- path: apps/*
template:
metadata:
name: "{{path.basename}}"
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/injunweb/k8s-resource.git
targetRevision: HEAD
path: "{{path}}"
destination:
server: https://kubernetes.default.svc
namespace: "{{path.basename}}"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true
- CreateNamespace=true
This ApplicationSet uses a Git generator to find all directories matching the apps/* pattern and automatically creates an ArgoCD Application for each directory. {{path.basename}} is a template variable that references the directory name, used as both the application name and namespace.
In practice, this configuration behaves like this:
- Directory Discovery: The Git generator finds all subdirectories under the
apps/directory. - Application Creation: Applies the template to each found directory to create an ArgoCD Application.
- Automatic Synchronization: Each created Application deploys the Helm charts or manifests from its directory to the cluster.
- Dynamic Management: When a new folder is added to the
apps/directory, a new Application is automatically created. When a folder is deleted, the corresponding Application is also automatically deleted.
Complete Workflow
Once that wiring was in place, the GitOps workflow in my homelab looked like this:
- Initial Bootstrap: When the administrator applies
app-of-apps.yamlto the cluster, the root Application is created. - First Synchronization: The root Application synchronizes the
app-of-appsrepository to create theinfra-apps-rootApplication. - Second Synchronization: The
infra-apps-rootApplication synchronizes theapplicationset.yamlfrom thek8s-resourcerepository to create the ApplicationSet. - Third Synchronization: The ApplicationSet creates individual Applications for each folder in the
apps/directory, and each Application deploys actual Kubernetes resources.
From that point on, adding a new folder under apps/ in the k8s-resource repository was enough for ArgoCD to detect and deploy a new application. Existing applications followed the same pattern: update the files in that directory, commit, and let ArgoCD reconcile the changes.
Conclusion
In this post, I installed ArgoCD on my homelab Kubernetes cluster and used the App of Apps pattern to put the GitOps workflow in place. From this point on, most of the cluster configuration lived in Git, and later pieces like storage, networking, and monitoring were all added on top of this structure.