EBS CSI driver and AL2023

After upgrading to Amazon Linux 2023 (AL2023) we started seeing errors from the aws-ebs-csi-driver running in our clusters.

ebs-plugin I0626 06:40:25.662215       1 main.go:154] "Initializing metadata"
ebs-plugin I0626 06:40:25.662374       1 metadata.go:66] "Attempting to retrieve instance metadata from IMDS"
ebs-plugin E0626 06:40:30.665263       1 metadata.go:72] "Retrieving IMDS metadata failed" err="could not get IMDS metadata: operation error ec2imds: GetInstanceIdentityDocument, canceled, context deadline exceeded"
ebs-plugin I0626 06:40:30.665357       1 metadata.go:75] "Attempting to retrieve instance metadata from Kubernetes API"

This is due to AL2023 improved security ensuring features blocking pods from calling metadata service on the nodes due to a network hop limit of 1. The aws-ebs-csi-driver eventually falls back to using the Kubernetes API, but we are waiting ~5 seconds for the call to timeout. With the release of aws-ebs-csi-driver v1.45.0 they have implemented a flag (--metadata-sources) allowing us to set a priority order or choose a specific way of getting metadata. In our case it would be set to "kubernetes".

This should prevent above shown errors.