<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[DevOps with Umar]]></title><description><![CDATA[DevOps with Umar]]></description><link>https://devopswithumar.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 14:25:03 GMT</lastBuildDate><atom:link href="https://devopswithumar.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[AWS NAT gateway cost: why it is so high and 5 ways to cut it]]></title><description><![CDATA[A NAT gateway costs $0.045 an hour plus $0.045 per GB. Most of that is avoidable: free S3 endpoints, fewer gateways, and image caching.]]></description><link>https://devopswithumar.hashnode.dev/aws-nat-gateway-cost-why-it-is-so-high-and-5-ways-to-cut-it</link><guid isPermaLink="true">https://devopswithumar.hashnode.dev/aws-nat-gateway-cost-why-it-is-so-high-and-5-ways-to-cut-it</guid><category><![CDATA[AWS]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Cost Optimization]]></category><dc:creator><![CDATA[Muhammad Umar]]></dc:creator><pubDate>Tue, 17 Nov 2026 15:57:33 GMT</pubDate><content:encoded><![CDATA[<p>In short:</p>
<ul>
<li>Each NAT gateway costs about $33 a month before any traffic, and every GB through it costs another $0.045.</li>
<li>Gateway endpoints for S3 and DynamoDB are free and take that traffic off the NAT gateway completely.</li>
<li>Dev and staging rarely need a NAT gateway per availability zone; production usually does, or a regional NAT gateway.</li>
</ul>
<p>A NAT gateway is one of the most common surprises on a startup's AWS bill. It looks harmless in the VPC diagram, and then Cost Explorer shows <strong>EC2-Other</strong> or <strong>NatGateway-Bytes</strong> as one of the top lines. The reason is that you pay twice: <strong>$0.045 per hour for each gateway, plus $0.045 for every GB of data it processes</strong> (us-east-1 prices, <a href="https://aws.amazon.com/vpc/pricing/">VPC pricing</a>).</p>
<p>Most of that money is avoidable. In the accounts I review, the biggest share of NAT traffic is usually going to AWS services that the VPC could reach for free. This post shows where NAT cost comes from, how to measure it, and five fixes in order of effort.</p>
<h2>What a NAT gateway actually costs</h2>
<p>Two charges, both in every region where you run one:</p>
<table>
<thead>
<tr>
<th>Charge</th>
<th>us-east-1 price</th>
<th>What it means</th>
</tr>
</thead>
<tbody><tr>
<td>Hourly</td>
<td>$0.045 per gateway hour</td>
<td>About $32.85 a month per gateway, even with zero traffic</td>
</tr>
<tr>
<td>Data processing</td>
<td>$0.045 per GB</td>
<td>Every GB in or out through the gateway, on top of normal data transfer</td>
</tr>
</tbody></table>
<p>The standard "high availability" layout puts one NAT gateway in each availability zone, so three zones means about $99 a month before a single byte moves. Then the traffic:</p>
<table>
<thead>
<tr>
<th>Monthly traffic through NAT</th>
<th>Processing charge</th>
<th>With 3 gateways</th>
</tr>
</thead>
<tbody><tr>
<td>100 GB</td>
<td>$4.50</td>
<td>about $103 a month</td>
</tr>
<tr>
<td>1 TB</td>
<td>$46</td>
<td>about $145 a month</td>
</tr>
<tr>
<td>10 TB</td>
<td>$461</td>
<td>about $560 a month</td>
</tr>
</tbody></table>
<p>And if the traffic leaves AWS for the internet, normal data transfer out is billed as well. A container that pulls a 1 GB image from a public registry 300 times a month moves 300 GB through NAT: about $13.50 in processing alone, before transfer.</p>
<h2>Step 1: find out what goes through your NAT gateways</h2>
<p>Do not guess. Start with the bill:</p>
<pre><code class="language-bash">aws ce get-cost-and-usage \
  --time-period Start=2026-10-01,End=2026-11-01 \
  --granularity MONTHLY --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["EC2 - Other"]}}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE \
  --query 'ResultsByTime[].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' \
  --output text | grep NatGateway
</code></pre>
<p>You will see lines like <code>USE1-NatGateway-Hours</code> and <code>USE1-NatGateway-Bytes</code>. If <code>Bytes</code> is bigger than <code>Hours</code>, traffic is the problem; if <code>Hours</code> dominates, you have more gateways than you need.</p>
<p>Then look at each gateway's traffic. This sums the bytes sent to destinations over the last 7 days:</p>
<pre><code class="language-bash">aws cloudwatch get-metric-statistics --namespace AWS/NATGateway \
  --metric-name BytesOutToDestination \
  --dimensions Name=NatGatewayId,Value=nat-0abc123def4567890 \
  --start-time "$(date -u -d '-7 days' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 604800 --statistics Sum
</code></pre>
<p>To see <em>where</em> the traffic goes, turn on VPC Flow Logs for the NAT gateway's network interface for a day or two and group by destination. The usual top destinations are S3, ECR, CloudWatch Logs, Docker Hub or GitHub, and your own services in another VPC.</p>
<h2>Fix 1: free gateway endpoints for S3 and DynamoDB</h2>
<p>This is the biggest and safest win. A <strong>gateway VPC endpoint</strong> for S3 or DynamoDB routes that traffic inside AWS, without the NAT gateway, and it has <strong>no hourly charge and no data charge</strong> (<a href="https://aws.amazon.com/vpc/pricing/">VPC pricing</a>).</p>
<p>In Terraform, one resource per service, attached to your private route tables:</p>
<pre><code class="language-hcl">resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id
}

resource "aws_vpc_endpoint" "dynamodb" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.dynamodb"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id
}
</code></pre>
<p>Nothing changes in your code: the SDK still calls the normal S3 endpoint, and the route table sends it through the gateway endpoint. If a bucket policy limits access by source IP, switch it to <code>aws:SourceVpce</code> first, because traffic through the endpoint no longer comes from the NAT gateway's public IP.</p>
<p>ECR stores image layers in S3, so the S3 gateway endpoint also removes most of the cost of pulling images from your own ECR repositories.</p>
<h2>Fix 2: interface endpoints for busy AWS services</h2>
<p>Other services, such as ECR's API, CloudWatch Logs, STS, Secrets Manager and SQS, need <strong>interface endpoints</strong>. These are not free: about $0.01 per hour per availability zone (about $7.30 a month per zone) plus $0.01 per GB processed.</p>
<p>They are worth it when a service moves a lot of data. Compare for one service across 3 zones:</p>
<table>
<thead>
<tr>
<th>Traffic to the service per month</th>
<th>Through NAT ($0.045/GB)</th>
<th>Interface endpoint (3 zones + $0.01/GB)</th>
</tr>
</thead>
<tbody><tr>
<td>50 GB</td>
<td>$2.25</td>
<td>$22.40</td>
</tr>
<tr>
<td>500 GB</td>
<td>$22.50</td>
<td>$26.90</td>
</tr>
<tr>
<td>2 TB</td>
<td>$92</td>
<td>$42</td>
</tr>
</tbody></table>
<p>So: add interface endpoints for the services that carry hundreds of GB a month (often CloudWatch Logs and ECR's <code>ecr.api</code> and <code>ecr.dkr</code> for clusters that scale a lot), and leave quiet services on NAT. The <a href="https://devopswithumar.com/tools/nat-gateway-cost-calculator/">NAT gateway cost calculator</a> does this comparison with your own numbers.</p>
<h2>Fix 3: stop pulling the same images and packages from the internet</h2>
<p>Every <code>docker pull</code> from Docker Hub or GitHub Container Registry, every <code>pip install</code> and <code>npm ci</code> on a private instance goes through NAT. Common fixes:</p>
<ul>
<li><strong>ECR pull-through cache.</strong> Point your clusters at ECR, which caches Docker Hub, GitHub and other public images the first time and serves them from your account after that. With the S3 gateway endpoint, repeat pulls stop touching NAT.</li>
<li><strong>Bigger, longer-lived nodes.</strong> Image layers are cached per node. Fifty small nodes that come and go every hour pull far more than ten that stay.</li>
<li><strong>Build in CI, not on the server.</strong> Instances that install packages at boot pay for that download every time they start.</li>
</ul>
<h2>Fix 4: right-size how many NAT gateways you run</h2>
<p>One NAT gateway per zone is the right answer for production: if a zone fails, the other zones keep their internet access. It is usually the wrong answer for dev, staging and preview environments.</p>
<table>
<thead>
<tr>
<th>Environment</th>
<th>Sensible layout</th>
<th>Monthly hourly cost</th>
</tr>
</thead>
<tbody><tr>
<td>Production, 3 zones</td>
<td>3 zonal NAT gateways, or one regional NAT gateway</td>
<td>about $99</td>
</tr>
<tr>
<td>Staging</td>
<td>1 NAT gateway shared by all zones</td>
<td>about $33</td>
</tr>
<tr>
<td>Dev, short-lived previews</td>
<td>1 NAT gateway, or a NAT instance</td>
<td>$3 to $33</td>
</tr>
</tbody></table>
<p>With one shared NAT gateway, traffic from the other zones crosses availability zones to reach it, which adds $0.01 per GB each way. For low-traffic environments that is far less than two extra gateways.</p>
<p><strong>What about the regional NAT gateway?</strong> Since November 2025, a NAT gateway can be created in <em>regional</em> mode (<a href="https://aws.amazon.com/about-aws/whats-new/2025/11/aws-nat-gateway-regional-availability">AWS announcement</a>). One regional NAT gateway serves every zone in the VPC, needs no public subnet, and expands to new zones on its own (<a href="https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-amazon-vpc-regional-nat-gateway">AWS blog</a>):</p>
<pre><code class="language-bash">aws ec2 create-nat-gateway --vpc-id vpc-0abc123def4567890 --availability-mode regional
</code></pre>
<p>It is billed per hour <strong>for each zone it runs in</strong>, at the same rate, plus the same per-GB charge. So it does not cut the bill by itself. What it removes is the work: one gateway ID, one route, no per-zone NAT setup to forget when you add a zone. For new production VPCs it is a good default; for an existing VPC that already works, move to it only when you are touching the network anyway.</p>
<h2>Fix 5: no NAT at all, where you can</h2>
<p>Some private workloads do not need the internet:</p>
<ul>
<li>A Lambda function or ECS task that only talks to S3, DynamoDB, SQS and Secrets Manager can use endpoints and skip NAT entirely.</li>
<li>Databases in private subnets almost never need outbound internet. Give them their own subnets with no route to NAT.</li>
<li>For tiny dev environments, a <strong>NAT instance</strong> on a <code>t4g.nano</code> costs a few dollars a month. The trade-off is real: you patch it, it is a single point of failure, and its bandwidth is small. Never use one for production.</li>
</ul>
<h2>What these fixes save: a real example</h2>
<p>A startup with one production VPC across 3 zones and a staging VPC across 3 zones, moving 3 TB a month through NAT. 2 TB of it is S3 and ECR layer traffic.</p>
<table>
<thead>
<tr>
<th>Item</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody><tr>
<td>Gateways</td>
<td>6 ($197)</td>
<td>4: 3 in production, 1 in staging ($131)</td>
</tr>
<tr>
<td>Data processed</td>
<td>3 TB ($138)</td>
<td>1 TB ($46)</td>
</tr>
<tr>
<td>S3 gateway endpoints</td>
<td>none</td>
<td>2 (free)</td>
</tr>
<tr>
<td><strong>Monthly NAT cost</strong></td>
<td><strong>about $335</strong></td>
<td><strong>about $177</strong></td>
</tr>
</tbody></table>
<p>That is about $1,900 a year back, from two free endpoints and one smaller staging layout, with no change to application code.</p>
<h2>NAT gateway cost checklist</h2>
<ol>
<li>Check the NAT lines in Cost Explorer: running hours and data processed.</li>
<li>Look at each gateway's traffic in CloudWatch, and turn on Flow Logs briefly to see destinations.</li>
<li>Add free gateway endpoints for S3 and DynamoDB to every VPC.</li>
<li>Add interface endpoints only for services that move hundreds of GB a month.</li>
<li>Cache container images in ECR instead of pulling from the internet.</li>
<li>Use one NAT gateway in dev and staging; keep one per zone, or one regional NAT gateway, in production.</li>
</ol>
<p>If your private subnets also hold leftovers that bill you every hour, see <a href="https://devopswithumar.com/blog/aws-charges-billing-you-for-nothing/">3 hidden AWS charges</a>. Planning a new VPC? Get the ranges right first with <a href="https://devopswithumar.com/blog/vpc-cidr-subnet-planning/">the VPC CIDR planning guide</a>. For a full review of your network and bill, see the <a href="https://devopswithumar.com/services/aws-cost-optimization/">AWS cost optimization service</a> or start with a <a href="https://devopswithumar.com/checkup/">Free AWS Review</a>.</p>
<p>Prices are from the <a href="https://aws.amazon.com/vpc/pricing/">Amazon VPC pricing page</a> and PrivateLink pricing for us-east-1, checked in November 2026. Other regions are slightly different.</p>
<h2>Common questions</h2>
<h3>Why is my NAT gateway so expensive?</h3>
<p>You pay twice: $0.045 per hour for each gateway (about $33 a month) and $0.045 for every GB it processes. Most of the surprise comes from the per-GB part, usually from S3, ECR image pulls or logs going through the NAT gateway when they could use VPC endpoints.</p>
<h3>Is a regional NAT gateway cheaper?</h3>
<p>Not by itself. A regional NAT gateway is billed per hour for each availability zone it runs in, at the same rate as a zonal one, plus the same per-GB charge. Its value is simpler setup and automatic expansion to new zones, not a lower price.</p>
<h3>Are S3 gateway endpoints really free?</h3>
<p>Yes. Gateway VPC endpoints for S3 and DynamoDB have no hourly charge and no data processing charge. Interface endpoints for other services do cost money.</p>
<h3>Can I run without a NAT gateway?</h3>
<p>Sometimes. If private workloads only talk to AWS services that have VPC endpoints, you can skip NAT entirely. For small dev environments, a NAT instance is cheaper but you manage and patch it yourself.</p>
<h3>How do I see how much traffic goes through my NAT gateway?</h3>
<p>Look at the BytesOutToDestination and BytesInFromDestination metrics in CloudWatch, or the NatGateway-Bytes usage type in Cost Explorer. VPC Flow Logs show which destinations the traffic goes to.</p>
<hr />
<p><em>Originally published on <a href="https://devopswithumar.com/blog/nat-gateway-cost/">DevOps with Umar</a>. Free tool for this: the <a href="https://devopswithumar.com/tools/nat-gateway-cost-calculator/">NAT gateway cost calculator</a>. More in <a href="https://devopswithumar.com/tools/">the tools list</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[EKS extended support: the 6x control plane bill and how to avoid it]]></title><description><![CDATA[EKS extended support costs $0.60 per cluster hour instead of $0.10, six times more. See which versions are affected and how to avoid the bill.]]></description><link>https://devopswithumar.hashnode.dev/eks-extended-support-the-6x-control-plane-bill-and-how-to-avoid-it</link><guid isPermaLink="true">https://devopswithumar.hashnode.dev/eks-extended-support-the-6x-control-plane-bill-and-how-to-avoid-it</guid><category><![CDATA[Kubernetes]]></category><category><![CDATA[AWS]]></category><category><![CDATA[Devops]]></category><dc:creator><![CDATA[Muhammad Umar]]></dc:creator><pubDate>Tue, 17 Nov 2026 15:22:30 GMT</pubDate><content:encoded><![CDATA[<p>In short:</p>
<ul>
<li>An EKS cluster on standard support costs $0.10 an hour (about $73 a month). On extended support it costs $0.60 an hour (about $438 a month).</li>
<li>Kubernetes 1.34 leaves standard support on 2 December 2026, and 1.33 already moved to extended support on 29 July 2026.</li>
<li>Set the upgrade policy to STANDARD on every cluster so an old version can never quietly start billing at the higher rate.</li>
</ul>
<p>EKS extended support has a simple price: <strong>$0.60 per cluster per hour instead of $0.10.</strong> That is about $438 a month for each cluster's control plane instead of about $73, six times more, for running a Kubernetes version that is more than 14 months old. Nothing about the cluster gets better. You are paying AWS to keep patching an old version for you.</p>
<p>This matters right now. Kubernetes 1.33 moved to extended support on 29 July 2026, and <strong>1.34 follows on 2 December 2026</strong>. If your startup created an EKS cluster in late 2025 and never upgraded it, it is about to cost six times more for the same thing.</p>
<p>This post shows how to find every cluster and its support status, stop the higher rate from ever starting by surprise, and upgrade with a plan you can repeat every few months.</p>
<h2>How EKS support and pricing work</h2>
<p>Every Kubernetes version on EKS goes through two phases (<a href="https://aws.amazon.com/blogs/containers/amazon-eks-extended-support-for-kubernetes-versions-pricing">AWS pricing post</a>):</p>
<table>
<thead>
<tr>
<th>Phase</th>
<th>How long</th>
<th>Control plane price</th>
</tr>
</thead>
<tbody><tr>
<td>Standard support</td>
<td>14 months after the version is released on EKS</td>
<td>$0.10 per cluster hour</td>
</tr>
<tr>
<td>Extended support</td>
<td>12 more months</td>
<td>$0.60 per cluster hour</td>
</tr>
<tr>
<td>End of extended support</td>
<td>after 26 months</td>
<td>AWS upgrades the cluster for you</td>
</tr>
</tbody></table>
<p>The charge is per cluster, not per node. A small startup with a production cluster, a staging cluster and a dev cluster on an old version pays about $1,095 a month more than it needs to. That is $13,000 a year for doing nothing.</p>
<p>Your worker nodes, Fargate pods, load balancers and EBS volumes are billed exactly the same in both phases. Only the control plane line changes, which is why it is easy to miss in Cost Explorer.</p>
<h2>The 2026 calendar</h2>
<p>These are the dates that matter for clusters running today (<a href="https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html">EKS version lifecycle</a>, mirrored on <a href="https://endoflife.date/amazon-eks">endoflife.date</a>):</p>
<table>
<thead>
<tr>
<th>Version</th>
<th>End of standard support</th>
<th>End of extended support</th>
</tr>
</thead>
<tbody><tr>
<td>1.32</td>
<td>23 March 2026 (already on extended)</td>
<td>23 March 2027</td>
</tr>
<tr>
<td>1.33</td>
<td>29 July 2026 (already on extended)</td>
<td>29 July 2027</td>
</tr>
<tr>
<td>1.34</td>
<td><strong>2 December 2026</strong></td>
<td>2 December 2027</td>
</tr>
<tr>
<td>1.35</td>
<td>27 March 2027</td>
<td>27 March 2028</td>
</tr>
<tr>
<td>1.36</td>
<td>2 August 2027</td>
<td>2 August 2028</td>
</tr>
<tr>
<td>1.37</td>
<td>1 December 2027</td>
<td>1 December 2028</td>
</tr>
</tbody></table>
<p>If you are on 1.32 or 1.33 today, you are already paying $0.60. If you are on 1.34, you have a few weeks. The safest target for an upgrade now is 1.35 or 1.36: both stay in standard support well into 2027.</p>
<h2>Step 1: find every cluster, its version and its support type</h2>
<p>Clusters live in regions, so check all of them. This prints each cluster with its region, version and upgrade policy:</p>
<pre><code class="language-bash">for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  for c in $(aws eks list-clusters --region "$r" --query 'clusters[]' --output text); do
    aws eks describe-cluster --region "$r" --name "$c" \
      --query '[cluster.name, cluster.version, cluster.upgradePolicy.supportType]' \
      --output text | sed "s/^/$r /"
  done
done
</code></pre>
<p>Example output:</p>
<pre><code class="language-text">us-east-1 prod-api 1.33 EXTENDED
us-east-1 staging 1.34 EXTENDED
eu-west-1 dev 1.35 STANDARD
</code></pre>
<p><code>EXTENDED</code> here means the cluster is allowed to go into extended support when its version gets old. It does not always mean it is paying the higher rate yet. Compare the version with the table above: <code>prod-api</code> on 1.33 is paying $0.60 today, and <code>staging</code> on 1.34 will start on 2 December.</p>
<h2>Step 2: make sure no cluster can drift into extended support</h2>
<p>Each cluster has an upgrade policy. The default lets a cluster move into extended support and the higher price. Setting it to <code>STANDARD</code> turns that off (<a href="https://docs.aws.amazon.com/eks/latest/userguide/disable-extended-support.html">AWS docs</a>):</p>
<pre><code class="language-bash">aws eks update-cluster-config --name staging --upgrade-policy supportType=STANDARD
</code></pre>
<p>With <code>STANDARD</code> set, a cluster that reaches the end of standard support is upgraded to the next version by AWS instead of being billed at $0.60. Two things to know before you rely on that:</p>
<ol>
<li><strong>You can only set it while the cluster is still on standard support.</strong> For a cluster already on extended support, the only way back to $0.10 is to upgrade it.</li>
<li><strong>An automatic upgrade happens on AWS's timing, not yours.</strong> It is a safety net against the bill, not an upgrade plan. Your workloads should already have been tested on the next version.</li>
</ol>
<p>In Terraform, set it on the cluster resource so new clusters get it from day one:</p>
<pre><code class="language-hcl">resource "aws_eks_cluster" "main" {
  name     = "prod-api"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.35"

  vpc_config {
    subnet_ids = var.private_subnet_ids
  }

  upgrade_policy {
    support_type = "STANDARD"
  }
}
</code></pre>
<p>The <code>upgrade_policy</code> block needs a recent AWS provider (it arrived in the 5.x series). Run <code>terraform plan</code> and check it shows <strong>update in-place</strong>, not a replacement of the cluster.</p>
<h2>Step 3: check what will break before you upgrade</h2>
<p>Most failed EKS upgrades are caused by something inside the cluster still using an API that the next version removed. EKS checks this for you with <strong>upgrade insights</strong>:</p>
<pre><code class="language-bash">aws eks list-insights --cluster-name prod-api \
  --query 'insights[].[name,insightStatus.status]' --output table
</code></pre>
<p>Any insight that is not <code>PASSING</code> needs work before the upgrade. To see details, including which clients called a deprecated API recently:</p>
<pre><code class="language-bash">aws eks describe-insight --cluster-name prod-api --id "$INSIGHT_ID"  # an id from list-insights
</code></pre>
<p>Also check these, in this order:</p>
<ul>
<li><strong>Add-ons.</strong> VPC CNI, CoreDNS, kube-proxy and the EBS CSI driver each have versions tied to the Kubernetes version. List the supported versions with <code>aws eks describe-addon-versions --kubernetes-version 1.35 --addon-name vpc-cni</code>.</li>
<li><strong>Helm charts you installed yourself.</strong> Ingress controllers, cert-manager, the AWS Load Balancer Controller and autoscalers like Karpenter all have a supported Kubernetes range in their release notes.</li>
<li><strong>Node AMIs.</strong> Managed node groups need an AMI for the new version. Amazon Linux 2 AMIs stopped at 1.32, so newer versions mean Amazon Linux 2023 or Bottlerocket nodes.</li>
</ul>
<h2>Step 4: upgrade one minor version at a time</h2>
<p>EKS only moves the control plane up one minor version per upgrade. Going from 1.33 to 1.35 is two rounds of this:</p>
<pre><code class="language-bash"># 1. Control plane (takes roughly 10 to 20 minutes, the API stays available)
aws eks update-cluster-version --name prod-api --kubernetes-version 1.34
aws eks wait cluster-active --name prod-api

# 2. Add-ons, to the version that matches 1.34 (ADDON_VERSION from describe-addon-versions)
aws eks update-addon --cluster-name prod-api --addon-name vpc-cni \
  --addon-version "$ADDON_VERSION" --resolve-conflicts PRESERVE

# 3. Managed node groups (nodes are replaced with a rolling update)
aws eks update-nodegroup-version --cluster-name prod-api --nodegroup-name default
</code></pre>
<p>Repeat for the next version. If Terraform manages the cluster, change <code>version</code> in Terraform instead, one step at a time, so the code and the cluster never disagree.</p>
<p>Three habits keep this boring:</p>
<ol>
<li><strong>Upgrade staging first</strong> and leave it on the new version for a few days of normal traffic.</li>
<li><strong>Check PodDisruptionBudgets</strong> before the node group update. A PDB that allows zero disruptions blocks the rolling update until it times out.</li>
<li><strong>Keep a note of what you did.</strong> You will do this again in about a year, and next time it takes half the time.</li>
</ol>
<h2>What the upgrade is worth</h2>
<p>For one cluster that is already on extended support:</p>
<table>
<thead>
<tr>
<th>Support</th>
<th>Hourly</th>
<th>Monthly (730 hours)</th>
<th>Yearly</th>
</tr>
</thead>
<tbody><tr>
<td>Extended support</td>
<td>$0.60</td>
<td>$438</td>
<td>$5,256</td>
</tr>
<tr>
<td>Standard support</td>
<td>$0.10</td>
<td>$73</td>
<td>$876</td>
</tr>
<tr>
<td><strong>Saving</strong></td>
<td>$0.50</td>
<td><strong>$365</strong></td>
<td><strong>$4,380</strong></td>
</tr>
</tbody></table>
<p>For three clusters, that is over $13,000 a year back, for roughly a day of planned work per cluster. Few cost fixes on AWS pay back faster. If you want to see the dates and cost for your own versions, the <a href="https://devopswithumar.com/tools/eks-version-support-checker/">EKS version support checker</a> does the maths.</p>
<h2>Stay out of extended support for good</h2>
<p>The calendar repeats: a new Kubernetes version arrives on EKS about every four months, and each one leaves standard support 14 months later. A simple routine keeps you ahead:</p>
<ul>
<li><strong>Every cluster has <code>supportType=STANDARD</code>.</strong> Put it in your Terraform module so new clusters get it automatically.</li>
<li><strong>Two upgrades a year.</strong> Pick two dates, for example March and September, and move every cluster to the newest version that has been on EKS for at least a month.</li>
<li><strong>A calendar reminder 60 days before each end-of-standard-support date</strong> for the versions you run.</li>
<li><strong>A budget alert on EKS.</strong> A jump from $73 to $438 per cluster is easy to spot if something is watching. See <a href="https://devopswithumar.com/blog/aws-monitoring-alerts-before-customers/">how to set up alerts that reach a person</a>.</li>
</ul>
<h2>EKS extended support checklist</h2>
<ol>
<li>List every cluster in every region with its version and upgrade policy.</li>
<li>Compare each version with the calendar and note which ones pay $0.60 now or soon.</li>
<li>Set <code>supportType=STANDARD</code> on every cluster still on standard support.</li>
<li>Run upgrade insights and fix anything not passing.</li>
<li>Upgrade staging one minor version at a time: control plane, add-ons, then nodes.</li>
<li>Repeat on production, and put the upgrade policy in your Terraform module.</li>
</ol>
<p>Running pods that restart for no clear reason after an upgrade? Start with <a href="https://devopswithumar.com/blog/kubernetes-mistakes-crash-pods/">3 Kubernetes mistakes that crash your pods</a>. If you would rather have someone plan and run the upgrade with you, the <a href="https://devopswithumar.com/services/kubernetes-eks/">EKS setup and upgrade service</a> covers it, or start with a <a href="https://devopswithumar.com/checkup/">Free AWS Review</a>.</p>
<p>Prices are from the <a href="https://aws.amazon.com/blogs/containers/amazon-eks-extended-support-for-kubernetes-versions-pricing">Amazon EKS extended support pricing post</a> and dates from the <a href="https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html">EKS Kubernetes version lifecycle</a>, checked in November 2026.</p>
<h2>Common questions</h2>
<h3>How much does EKS extended support cost?</h3>
<p>$0.60 per cluster per hour, against $0.10 on standard support. That is about $438 a month per cluster instead of about $73. Worker nodes, Fargate and EBS are billed the same either way.</p>
<h3>When does Kubernetes 1.34 leave standard support on EKS?</h3>
<p>On 2 December 2026. From that day a 1.34 cluster bills at the extended support rate unless you upgrade it or its upgrade policy is set to STANDARD.</p>
<h3>Can I turn off extended support for a cluster that is already on it?</h3>
<p>No. You can only change the upgrade policy while the cluster is still on standard support. A cluster already on extended support goes back to $0.10 an hour once you upgrade it to a version in standard support.</p>
<h3>What happens if I set the upgrade policy to STANDARD and forget to upgrade?</h3>
<p>AWS upgrades the control plane to the next Kubernetes version when standard support ends. That avoids the extra charge, but your workloads meet a new version on AWS's schedule instead of yours, so plan the upgrade before that date.</p>
<h3>Do I have to upgrade one version at a time?</h3>
<p>Yes. EKS upgrades the control plane one minor version at a time, so 1.32 to 1.35 means three upgrades. Nodes and add-ons should follow each step.</p>
<hr />
<p><em>Originally published on <a href="https://devopswithumar.com/blog/eks-extended-support-cost/">DevOps with Umar</a>. Free tool for this: the <a href="https://devopswithumar.com/tools/eks-version-support-checker/">EKS version support checker</a>. More in <a href="https://devopswithumar.com/tools/">the tools list</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[How to unlock Terraform state (force-unlock) and recover it]]></title><description><![CDATA[Stuck on "Error acquiring the state lock"? Find the lock ID, run terraform force-unlock safely, and roll back a bad state from S3 versioning.]]></description><link>https://devopswithumar.hashnode.dev/how-to-unlock-terraform-state-force-unlock-and-recover-it</link><guid isPermaLink="true">https://devopswithumar.hashnode.dev/how-to-unlock-terraform-state-force-unlock-and-recover-it</guid><category><![CDATA[Terraform]]></category><category><![CDATA[Devops]]></category><category><![CDATA[AWS]]></category><dc:creator><![CDATA[Muhammad Umar]]></dc:creator><pubDate>Sun, 15 Nov 2026 17:57:00 GMT</pubDate><content:encoded><![CDATA[<p>In short:</p>
<ul>
<li>Read the lock ID and the Who/Created lines in the error first. Only force-unlock when no apply is still running.</li>
<li>Unlock with terraform force-unlock LOCK_ID, never by editing or deleting the state file.</li>
<li>A versioned S3 bucket turns a corrupted state into a two-minute rollback.</li>
</ul>
<p>To unlock Terraform state, copy the lock ID from the error message, make sure no other <code>terraform apply</code> is still running, and run <code>terraform force-unlock LOCK_ID</code>. That is the whole fix for most stuck locks. The rest of this post shows how to find the ID, when unlocking is safe, what to do when the lock lives in S3 or DynamoDB, and how to recover if the state itself is damaged.</p>
<p>If you have not set up a remote backend yet, start with <a href="https://devopswithumar.com/blog/terraform-remote-state-s3-kms/">Terraform remote state on S3 with KMS and locking</a>. This post is for when that setup is already running and something went wrong.</p>
<h2>Why Terraform locks the state</h2>
<p>Every <code>plan</code> and <code>apply</code> takes a lock on the state before it reads it. The lock stops two runs from writing the same state at the same time, which is how state gets corrupted. When the run finishes, Terraform removes the lock.</p>
<p>A lock gets "stuck" when the run that took it never finished cleanly:</p>
<ul>
<li>you pressed Ctrl+C during an apply, or your laptop went to sleep</li>
<li>a CI job was cancelled or timed out halfway</li>
<li>the network dropped while Terraform was writing state</li>
</ul>
<p>The next run then fails with this:</p>
<pre><code class="language-text">Error: Error acquiring the state lock

Lock Info:
  ID:        6f1c2a9e-0b3d-4c7e-9a51-2d8f4e7b1c90
  Path:      my-company-tfstate/prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@fv-az123-456
  Version:   1.12.2
  Created:   2026-10-31 09:14:02 UTC
</code></pre>
<h2>Step 1: get the lock ID and check who holds it</h2>
<p>The <code>ID</code> line is the lock ID you need. Before you unlock anything, read the <code>Who</code> and <code>Created</code> lines:</p>
<ul>
<li><strong>Who</strong> tells you which machine or CI runner took the lock.</li>
<li><strong>Created</strong> tells you when.</li>
</ul>
<p>If the lock is a few seconds old, someone is probably applying right now. Wait. If it is from a CI job you can see was cancelled, or from your own terminal that crashed an hour ago, it is stale and safe to remove.</p>
<p>When a lock is only busy, not stuck, you do not need to unlock at all. Tell Terraform to wait for it instead:</p>
<pre><code class="language-bash">terraform apply -lock-timeout=5m
</code></pre>
<h2>Step 2: unlock Terraform state with force-unlock</h2>
<p>From the same folder (so Terraform uses the same backend), run:</p>
<pre><code class="language-bash">terraform force-unlock 6f1c2a9e-0b3d-4c7e-9a51-2d8f4e7b1c90
</code></pre>
<p>Terraform asks you to confirm, then removes the lock. Add <code>-force</code> to skip the question in a script, but only when you are certain nothing else is running.</p>
<p>Then run <code>terraform plan</code> before anything else. If the interrupted apply changed some resources but did not finish, the plan shows you what is left.</p>
<p><strong>Never unlock by deleting or editing <code>terraform.tfstate</code>.</strong> The lock is separate from the state. Touching the state file does not remove the lock, and it can destroy the record of your infrastructure.</p>
<h2>Where the lock lives (if force-unlock cannot reach it)</h2>
<p><code>force-unlock</code> works for almost everyone. If it fails, for example because the CI role that took the lock is the only one allowed to delete it, you can remove the lock where it is stored.</p>
<p><strong>S3 native locking (<code>use_lockfile = true</code>, Terraform 1.10 and later).</strong> The lock is a small object next to your state, with <code>.tflock</code> added to the key. You can read it to see the lock ID and owner:</p>
<pre><code class="language-bash">aws s3 cp s3://my-company-tfstate/prod/terraform.tfstate.tflock -
</code></pre>
<p>Deleting that object removes the lock. Use this only after the same checks as Step 1.</p>
<p><strong>DynamoDB locking (the older setup).</strong> The lock is an item in your lock table whose <code>LockID</code> is <code>bucket/key</code>, for example <code>my-company-tfstate/prod/terraform.tfstate</code>. Delete that one item to remove the lock. Leave the item ending in <code>-md5</code> alone: it is a checksum of the state, not a lock.</p>
<h2>Step 3: recover a damaged state from S3 versioning</h2>
<p>Sometimes the interrupted run did more than leave a lock. If <code>terraform plan</code> now wants to recreate resources that clearly exist, the state was damaged. With versioning on the bucket, you can roll back:</p>
<ol>
<li>List the versions of the state file:<pre><code class="language-bash">aws s3api list-object-versions --bucket my-company-tfstate --prefix prod/terraform.tfstate
</code></pre>
</li>
<li>Pick the last version from before the problem (check <code>LastModified</code>).</li>
<li>Download it and copy it back as the current version:<pre><code class="language-bash">aws s3api get-object --bucket my-company-tfstate --key prod/terraform.tfstate \
  --version-id &lt;VERSION_ID&gt; good.tfstate
aws s3 cp good.tfstate s3://my-company-tfstate/prod/terraform.tfstate
</code></pre>
</li>
<li>Run <code>terraform plan</code> and check that it shows what you expect.</li>
</ol>
<p>Before any risky state work, keep a local copy too: <code>terraform state pull &gt; backup.tfstate</code>. It costs one second.</p>
<p>If the bucket has no versioning, you cannot roll back. You will be using <code>terraform import</code> to rebuild the state one resource at a time. Turn versioning on today.</p>
<h2>"Reset" Terraform state: what people usually mean</h2>
<p>There is no single reset command, and that is a good thing. Pick the one that matches the problem:</p>
<ul>
<li><strong>Terraform points at the wrong backend:</strong> <code>terraform init -reconfigure</code>.</li>
<li><strong>A resource was deleted by hand and Terraform still tracks it:</strong> <code>terraform state rm &lt;address&gt;</code>.</li>
<li><strong>A resource exists but Terraform does not know about it:</strong> <code>terraform import &lt;address&gt; &lt;id&gt;</code>, or an <code>import</code> block in your code.</li>
</ul>
<p>Never delete the whole state to "start fresh" on real infrastructure. Terraform would try to create everything again.</p>
<h2>Stop it happening again</h2>
<ul>
<li><strong>In CI, do not let two runs overlap.</strong> In GitHub Actions, put a <code>concurrency</code> group on the Terraform job so a second push waits instead of fighting for the lock.</li>
<li><strong>Do not cancel a running apply.</strong> Let it finish, then fix forward.</li>
<li><strong>Keep state remote, locked, versioned and encrypted.</strong> The full setup is in my <a href="https://devopswithumar.com/blog/terraform-remote-state-s3-kms/">remote state on S3 with KMS</a> post.</li>
</ul>
<h2>Checklist</h2>
<ol>
<li>Read the lock ID, Who and Created lines before doing anything.</li>
<li>Busy lock: wait with <code>-lock-timeout</code>. Stale lock: <code>terraform force-unlock LOCK_ID</code>.</li>
<li>Run <code>terraform plan</code> straight after unlocking.</li>
<li>Damaged state: roll back the last good version from S3 versioning.</li>
<li>Add a <code>concurrency</code> group to CI so this does not happen again.</li>
</ol>
<p>Short on time? The <a href="https://www.youtube.com/shorts/sKhXlfJYhkI">45-second version</a> covers the three state fixes. HashiCorp documents every option in the <a href="https://developer.hashicorp.com/terraform/cli/commands/force-unlock">force-unlock command reference</a>. For more traps, read <a href="https://devopswithumar.com/blog/terraform-mistakes-that-cost-hours/">3 Terraform mistakes that cost me hours</a>. And if you want someone to look over your Terraform setup, see <a href="https://devopswithumar.com/services/terraform-consultant/">Terraform consulting</a>.</p>
<p><a class="embed-card" href="https://www.youtube.com/watch?v=sKhXlfJYhkI">https://www.youtube.com/watch?v=sKhXlfJYhkI</a></p>

<hr />
<p><em>Originally published on <a href="https://devopswithumar.com/blog/terraform-state-lock-version-recover/">DevOps with Umar</a>. I keep free browser tools for this kind of work there: an <a href="https://devopswithumar.com/tools/aws-cost-calculator/">AWS savings estimator</a>, a <a href="https://devopswithumar.com/tools/cron-generator/">cron expression generator</a> and more in <a href="https://devopswithumar.com/tools/">the tools list</a>.</em></p>
]]></content:encoded></item></channel></rss>