Skip to content

Commit 36c474d

Browse files
nitrocodeclaude
nitrocode
and
claude
committed
docs(blog): add draft for pruning unused resources post
Multi-resource cleanup post covering the c7n-org pipeline. Topics: 4-stage state machine (tag, resurrect, quarantine, delete) running on an hourly GitHub Action cron, rate-limited deletion, eight resource types (IAM roles, security groups, EIPs, EBS volumes, RDS instances, ELBs, snapshots, NAT gateways), and an estimated cost-impact table. Includes a mermaid state diagram, an IAM role deep dive, and an audit side-benefit subsection. Co-Authored-By: AI pair (📡 RB's pair) <noreply@anthropic.com>
1 parent 1f42fa4 commit 36c474d

1 file changed

Lines changed: 364 additions & 0 deletions

File tree

‎_drafts/2025-03-15-pruning-unused-resources.md‎

Lines changed: 364 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,364 @@
1+
---
2+
layout: post
3+
title: "Pruning the Cloud: Deleting Unused Resources"
4+
description: "A single Cloud Custodian pipeline that has deleted 10,000 unused IAM roles, security groups, elastic IPs, EBS volumes, RDS instances, ELBs, snapshots, and NAT gateways, and continues to delete more every day. One pattern, eight resource types, rate-limited so mistakes are bounded."
5+
date: 2025-03-15
6+
tags: [Security, AWS, Cloud Custodian, Cloud, Infrastructure]
7+
permalink: /blog/pruning-unused-resources/
8+
---
9+
10+
## TL;DR
11+
12+
I built a Cloud Custodian pipeline that has deleted roughly 10,000 unused
13+
resources across a multi-account cloud estate and continues to delete more
14+
every day. The same four-stage state machine (tag, resurrect, quarantine,
15+
delete) handled eight resource types: IAM roles, security groups, elastic
16+
IPs, EBS volumes, RDS instances, ELBs, old snapshots, and NAT gateways.
17+
The filters change per resource type. The shape does not.
18+
19+
The most interesting case was IAM roles, which get a deep dive below. The
20+
short version: about one in five was unused, and the dangerous category
21+
was roles whose trust policies pointed at external accounts the org no
22+
longer engaged with. CSPM tools do not flag those.
23+
24+
If you take one thing from this post: manual cleanups are events; the
25+
pipeline is the control. Cloud Custodian's combination of filters, tags,
26+
and rate-limited actions is well-shaped for "find unused resources of
27+
type X" problems, and the second pipeline is mostly a copy-paste of the
28+
first.
29+
30+
| Outcome | What it means |
31+
|---|---|
32+
| ~10,000 unused resources deleted and counting | Continuous cleanup, not a one-time audit |
33+
| 8 resource types covered: IAM roles, SGs, EIPs, EBS, RDS, ELBs, snapshots, NAT gateways | One pattern, many resource types |
34+
| Rate-limited deletions: only a fraction of tagged resources per run | Mistakes are bounded; rollback is feasible |
35+
| Audit-flagged high-privilege roles cleaned up as a side effect | Shorter, defensible administrator list at next review |
36+
| c7n-org runs multi-account on an hourly GitHub Action cron | Hygiene is infrastructure-as-code; no persistent infra to maintain |
37+
38+
## Why this matters
39+
40+
Cloud estates grow outward. New accounts, new buckets, new roles, new
41+
compute. Each addition has a clear owner on day one. A few years later, a
42+
large chunk of the surface area is dead. Unused IAM roles, security groups
43+
attached to nothing, EBS volumes detached from any instance, RDS instances
44+
nobody connects to, NAT gateways racking up charges for traffic that no
45+
longer flows.
46+
47+
Most CSPM tools score resources by configuration: are the security group
48+
rules valid, is the IAM trust policy syntactically correct, does the
49+
snapshot have encryption. A security group with one ingress rule on port
50+
22 to `0.0.0.0/0` is scored correctly. A security group attached to
51+
nothing, with the same rule, is also scored correctly. The configuration
52+
is identical. The difference is whether it is in use.
53+
54+
The unused-and-forgotten category is invisible to compliance tooling. It
55+
is also where most of the abandoned attack surface lives. Each unused IAM
56+
role is a dormant pivot. Each unused security group with a stale rule is
57+
a misconfiguration waiting to be re-attached to something. Each abandoned
58+
snapshot is data nobody is tracking. Each unused NAT gateway is a
59+
recurring bill and an unmonitored egress path.
60+
61+
The headline risk is "too many resources." The real risk is the long tail
62+
nobody can defend.
63+
64+
## What I did
65+
66+
**Goal**: build automation that continuously cleans up unused resources
67+
across a multi-account cloud estate without breaking anything still in
68+
use.
69+
70+
Built a Cloud Custodian pipeline that runs across every AWS account in
71+
the org and looks for unused resources across eight types:
72+
73+
- **IAM roles**: no principal calls in the last 330 days
74+
- **Security groups**: not attached to any ENI, with exceptions for `default`
75+
and Kubernetes-managed patterns
76+
- **Elastic IPs**: unattached
77+
- **EBS volumes**: detached
78+
- **RDS instances**: no recent connections (database engine metric)
79+
- **ELBs**: classic, application, and network load balancers with no targets
80+
- **EBS snapshots**: orphaned (source volume deleted) and older than threshold
81+
- **NAT gateways**: in subnets with no active workloads routing through them
82+
83+
Each resource type has its own policy, but the shape is the same: tag
84+
when first found, remove the tag if the resource comes back into use,
85+
quarantine after a buffer period, delete after a further buffer. The
86+
thresholds vary (security groups can move faster than IAM roles; EIPs go
87+
quickly; RDS deserves a longer buffer).
88+
89+
Cumulative pipeline output: about 10,000 deletions across all resource
90+
types since deployment, and the number continues to grow every day.
91+
92+
## How I did it: a four-stage state machine in Cloud Custodian
93+
94+
I used [Cloud Custodian](https://cloudcustodian.io/) (`c7n`) with
95+
[c7n-org](https://cloudcustodian.io/docs/tools/c7n-org.html) for
96+
multi-account orchestration. The whole thing runs in a GitHub Action
97+
on an hourly cron schedule. No persistent infrastructure, no Lambda,
98+
no dedicated cron host. The runner spins up, evaluates the policies
99+
across every account in the org, takes actions, posts notifications to
100+
Slack, and exits.
101+
102+
The pipeline is a four-stage state machine where the state is encoded
103+
as AWS tags on each resource. No external database, no separate
104+
workflow engine. Tags do the work.
105+
106+
```mermaid
107+
stateDiagram-v2
108+
[*] --> Untagged: resource created
109+
Untagged --> Tagged: unused > 330d (hourly)
110+
Tagged --> Untagged: resurrected
111+
Tagged --> Quarantined: tagged > 31d, policies detached (daily)
112+
Quarantined --> Untagged: resurrected
113+
Quarantined --> Deleted: quarantined > 7d (daily)
114+
Deleted --> [*]
115+
```
116+
117+
The resurrect transition is the safety net. Anyone who exercises the
118+
resource at any stage before deletion bumps it back to Untagged and
119+
starts the inactivity clock over.
120+
121+
The four stages, using IAM roles as the example:
122+
123+
### Stage 1: tag (hourly)
124+
125+
Find resources matching the "unused" criteria, exclude well-known
126+
exceptions, tag them with `cc-unused-found-date: {now}`.
127+
128+
```yaml
129+
- name: iam-role-unused-tag
130+
resource: iam-role
131+
description: Tag IAM roles unused for more than 330 days
132+
conditions:
133+
# IAM is a global service. Only run in us-east-1 to avoid
134+
# one redundant pass per region.
135+
- region: us-east-1
136+
filters:
137+
- "tag:cc-unused-found-date": absent
138+
- "tag:cc-ignore": absent
139+
- or:
140+
- and:
141+
- 'RoleLastUsed.LastUsedDate': empty
142+
- type: value
143+
key: CreateDate
144+
op: greater-than
145+
value_type: age
146+
value: 330
147+
- type: value
148+
key: 'RoleLastUsed.LastUsedDate'
149+
value_type: age
150+
value: 330
151+
op: greater-than
152+
- not:
153+
- or:
154+
- type: value
155+
key: 'RoleName'
156+
op: regex
157+
value: '^AWSReservedSSO_.*'
158+
- type: value
159+
key: 'Arn'
160+
op: contains
161+
value: '/aws-service-role/'
162+
actions:
163+
- type: tag
164+
tags:
165+
cc-unused-found-date: "{now}"
166+
```
167+
168+
### Stage 2: resurrect (hourly)
169+
170+
If a tagged resource becomes active again or turns out to be recently
171+
created, remove the tag. This is the safety net. Anyone who exercises the
172+
resource during the quarantine window buys it another full inactivity
173+
threshold.
174+
175+
### Stage 3: quarantine (daily, after the buffer period)
176+
177+
If a resource has been tagged for more than the buffer period and no
178+
resurrection happened, neuter it without deleting it. The mechanism
179+
depends on the resource type:
180+
181+
- **IAM role**: detach every attached policy. The role still exists; it
182+
just cannot do anything.
183+
- **Security group**: revoke every ingress and egress rule.
184+
- **EBS volume / snapshot**: tag for deletion; lifecycle policies handle
185+
the actual removal.
186+
- **RDS**: stop the instance.
187+
- **NAT gateway / EIP / ELB**: usually skipped (move directly to delete
188+
after the longer inactivity threshold).
189+
190+
Tag with `cc-detach-policy-date: {now}` and post a Slack notification.
191+
192+
This is the most important step. A loud failure during quarantine (an
193+
`AccessDenied` in someone's CI logs, a `ConnectionRefused` from a stopped
194+
RDS) is recoverable. A silent deletion of a still-used resource is not.
195+
196+
### Stage 4: delete (daily, after a further buffer)
197+
198+
If the resource has been in the quarantine state without resurrection,
199+
delete it. Post another Slack notification.
200+
201+
For IAM roles, the total quarantine window is roughly 38 days on top of
202+
the 330-day inactivity threshold. A role has to be genuinely abandoned
203+
for over a year before deletion.
204+
205+
### Rate-limited deletion
206+
207+
A Cloud Custodian `reduce` filter caps the number of resources processed
208+
per run. Without it, the first run after deployment would attempt to
209+
delete every tagged resource at once. With it, the deletion is throttled
210+
to a controlled fraction per run, spreading any unintended consequences
211+
across days instead of minutes.
212+
213+
```yaml
214+
- type: reduce
215+
order: randomize
216+
limit-percent: 5
217+
```
218+
219+
If a percent is set to 5, a fully-tagged backlog of a thousand resources
220+
takes about twenty days to fully process. Anyone affected has a long
221+
window to surface a complaint before the next batch goes through. When
222+
something does break, the blast radius is one batch, not the entire
223+
backlog.
224+
225+
### Why tag-based state is the right model
226+
227+
The pipeline has no external state store. Each resource's progression
228+
through the stages is encoded in tags on the resource itself. AWS APIs
229+
are authoritative; the state machine cannot drift away from reality. If
230+
someone manually removes a tag, the resource re-enters the pipeline from
231+
the start. If someone manually deletes a resource, the pipeline never
232+
sees it again.
233+
234+
Slack notifications at quarantine and delete give operators a continuous,
235+
low-noise view of what the pipeline is doing without anyone needing to
236+
open a dashboard.
237+
238+
## Deep dive: the IAM role case
239+
240+
IAM roles deserve their own section because the failure mode is more
241+
interesting than "this resource is unused, delete it."
242+
243+
About one in five IAM roles in the org was unused. Most were boring:
244+
vendor integrations that ended, personal AWS accounts of engineers who
245+
had left, third-party tools the org no longer paid for. The first
246+
thousand deletes were fast.
247+
248+
The interesting category was roles with trust policies pointing at
249+
external accounts the org no longer engaged with. The role itself was
250+
technically perfect: a syntactically valid trust policy, IAM conditions
251+
in place, no wildcards on resource. But the AWS account ID in the trust
252+
policy was for a vendor we stopped using two years ago. The vendor has
253+
long since cycled out their AWS accounts. The role is one credential
254+
leak away from being someone else's pivot into the cloud estate.
255+
256+
CSPM tools do not flag this. The configuration is correct. The intent is
257+
gone.
258+
259+
Pruning these required cross-referencing every role's trust policy
260+
against the vendor inventory and the departed-employee list. The first
261+
pass has to happen by hand. After that, you can maintain a list of
262+
"external accounts we still trust" and let the policy flag anything
263+
outside it.
264+
265+
### Side benefit: shorter list for the auditors
266+
267+
The cleanup had a downstream benefit nobody planned for. Compliance and
268+
external auditors had flagged a recurring set of high-privilege roles
269+
during prior reviews. Most of those were not roles people were actively
270+
using; they were roles attached to systems that no longer ran. The
271+
pipeline either deleted them outright or surfaced them for explicit
272+
downscoping as part of the broader review.
273+
274+
By the next audit cycle, the list of administrators had three
275+
categories: currently used roles, CI/CD roles, and a small number of
276+
break-glass roles. Each easy to justify. The category that auditors care
277+
most about, and engineers can least defend ("why does this dormant role
278+
have admin?"), no longer existed.
279+
280+
## Cost impact
281+
282+
IAM roles and security groups are free; deleting them does not reduce
283+
the AWS bill. The other six categories do. Using estimated counts based
284+
on the 10,000-resource total and public AWS list prices (us-east-1):
285+
286+
| Resource type | Estimated count | Annual list price each | Estimated annual savings |
287+
|---|---|---|---|
288+
| IAM roles | ~5,000 | $0 | $0 |
289+
| Security groups | ~2,500 | $0 | $0 |
290+
| EBS snapshots (~100 GB avg) | ~1,500 | ~$60 | ~$90k |
291+
| EBS volumes (~200 GB avg, gp3) | ~500 | ~$192 | ~$96k |
292+
| Elastic IPs (unattached) | ~200 | ~$44 | ~$9k |
293+
| ELBs (base hourly) | ~200 | ~$197 | ~$39k |
294+
| RDS instances (small-tier avg) | ~50 | ~$840 | ~$42k |
295+
| NAT gateways (base hourly) | ~50 | ~$394 | ~$20k |
296+
| **Total** | **~10,000** | | **~$300k/year** |
297+
298+
A few caveats. These are list-price estimates, not invoice numbers.
299+
Reserved instance commitments, savings plans, and inter-region
300+
variations all bend real spend lower. The math is also
301+
order-of-magnitude across a long tail of resource sizes, not a precise
302+
audit. The point is the shape: most of the savings come from a small
303+
number of expensive resources (RDS, NAT gateways) and a long tail of
304+
medium-priced ones (volumes, snapshots), not from the bulk of the
305+
cleanup (IAM roles and security groups).
306+
307+
This was not the reason to do the cleanup. The security argument
308+
(dormant attack surface) and the operational argument (continuous
309+
hygiene instead of one-off audits) are the load-bearing ones. The cost
310+
recovery is a side benefit that makes the project easier to justify in
311+
budget review.
312+
313+
## What I would do differently
314+
315+
**Ship the pipeline first in dry-run, let it produce the audit candidate
316+
list.** The audit gave me confidence in the approach. The pipeline is
317+
what does the work. Building them in that order made sense at the time
318+
but reversed the leverage: I spent audit hours hand-rolling queries the
319+
pipeline would have produced for free. Deploy the pipeline in dry-run
320+
from day one, review the candidates in batches, then enable the
321+
quarantine and delete actions once the candidate list looks right.
322+
323+
**Add the rate limiter before the first delete, not after the first
324+
incident.** I learned the value of the `reduce` filter the way these
325+
things are usually learned: by deleting too many resources too fast on a
326+
non-prod account. Set the percent low from day one; you can raise it
327+
once the operational confidence is there.
328+
329+
**Lean harder on c7n's built-in patterns.** Two specifics. First, use
330+
[`mark-for-op`](https://cloudcustodian.io/docs/aws/resources/iam-role.html#actions)
331+
to encode both the operation and the scheduled date in a single tag
332+
value (`c7n_status: detach-policy@2025-04-15`) instead of custom date
333+
tags like `cc-unused-found-date`. The mark-for-op convention makes
334+
filter logic shorter and the operator intent self-documenting. Second,
335+
when rate-limiting via `reduce`, use
336+
`sort-by: 'RoleLastUsed.LastUsedDate'` with `value_type: date` and
337+
`sort-order: asc` instead of `order: randomize`. Oldest-unused-first
338+
deletes the highest-confidence candidates first; random order will
339+
occasionally delete a borderline role before a five-year-dormant one.
340+
341+
**Tag enforcement at resource creation, not just at cleanup time.** The
342+
least productive hour of any cleanup is the one spent chasing down who
343+
owns a resource created by an engineer who left two years ago. Require
344+
owner tags via service control policies or Terraform module conventions
345+
at creation time, and the cleanup pipeline can route notifications to
346+
the actual owner instead of guessing.
347+
348+
## Takeaways
349+
350+
- **Manual cleanups are events; the pipeline is the control.** A
351+
one-time audit gets the estate to a defensible baseline. Continuous
352+
automation keeps it there.
353+
- **Tags as state are durable.** No external store, no drift, no
354+
separate workflow engine. The cloud APIs are authoritative.
355+
- **Quarantine by neutering, not by deleting.** Detach policies, revoke
356+
rules, stop instances. If someone tries to use the resource, they get
357+
a loud, recoverable failure rather than a silent disappearance.
358+
- **The pattern generalizes.** The same four-stage shape extends almost
359+
unchanged across IAM roles, security groups, EIPs, EBS volumes, RDS,
360+
ELBs, snapshots, and NAT gateways. Write the first pipeline well; the
361+
rest are filter swaps.
362+
- **CSPM scores configuration; pipelines score intent.** A role whose
363+
intent has expired is invisible to compliance tooling. Continuous
364+
cleanup is how that gap closes.

0 commit comments

Comments
 (0)