AWS IAM Policy Troubleshooting: Why Access Is Denied
#AccessDenied Is a Verdict, Not a Diagnosis#link
The most frustrating part of an AWS AccessDenied error is that it tells you nothing about which layer produced it. You can be staring at an identity policy that grants s3:GetObject on exactly the right ARN, with no Deny statement anywhere in the document, and still get a 403. The denial came from somewhere your identity policy cannot see: a service control policy, a permission boundary, a KMS key policy on the object's encryption key, or a VPC endpoint policy. IAM evaluates six separate policy types and returns one flat error when any of them says no.
TL;DR: Read the evaluation chain in order, confirm which layer denied you using CloudTrail and IAM Access Analyzer rather than guessing, and remember that simulate-principal-policy silently ignores resource-based policies unless you pass one in. Most stubborn AccessDenied cases are cross-account access missing one of its two required grants, or an S3 action that needs a KMS key policy grant on top of an IAM grant.
The First Question: Which Layer Said No?
AWS evaluates authorization in a fixed order, and the order matters because an explicit Deny at any level wins immediately. A Deny inside a service control policy overrides a full-allow identity policy, a resource policy Allow, and a permissions boundary, all at once. If you are debugging from the identity policy upward, you are debugging in the wrong direction.
The practical consequence of that diagram is that the layer you are looking at is often not the layer that denied you. The permission boundary is the clearest example: it can never grant anything, only limit, so a boundary that omits an action produces a denial even when a permissive identity policy sits right next to it. Service control policies behave the same way for every account in an organization, with the notable exception that they do not apply to the organization's management account.
The error code you get is a hint about which service rejected you, and it is not consistent across AWS. S3 returns AccessDenied, EC2 returns UnauthorizedOperation, KMS and Lambda return AccessDeniedException, and IAM and STS return AccessDenied. If you are grepping logs for the string AccessDenied across all services, you will silently miss every EC2 denial.
Reading the Denial Instead of Guessing At It
The fastest way to stop guessing is to make the failing call with --debug and read the raw request and response. The AWS CLI prints the full HTTP exchange, including the service's own error XML or JSON, which frequently names the specific permission that was missing. S3 in particular returns a <Code> element and, for signature or policy issues, an explanatory message that never reaches the human-readable error line.
Raw S3 403 response for a GetObject call
The CLI's one-line error says only AccessDenied. The response body identifies the request so you can match it to a CloudTrail event.
<?xml version="1.0" encoding="UTF-8"?> <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>3E5C1A2B9F4D7E01</RequestId> <HostId>kQ9v2mNpLxT4rB8sYwZ6cJdHfEgA1uVo=</HostId> </Error>
S3 deliberately returns 403 AccessDenied instead of 404 NoSuchKey when the caller lacks s3:ListBucket on the bucket. The service refuses to confirm whether an object exists to a principal that is not allowed to enumerate. If you are chasing a 403 on an object you are certain exists, check ListBucket first, not GetObject.
For a denial that already happened, IAM Access Analyzer's CloudTrail-based findings and the console's Access Denied explainer are the only tools that reverse-engineer the specific policy statement responsible. They are worth the console trip: the explainer names the policy type, the statement, and the condition that failed, which is information no amount of reading JSON will give you reliably when conditions are involved.
Why simulate-principal-policy Keeps Lying to You
The simulation API is the right instinct and the wrong default. Out of the box it evaluates the identity policies attached to the principal, and that is roughly it. It does not consider a resource-based policy unless you hand it one, and it does not consider a permissions boundary unless you pass that too. Teams routinely conclude 'the simulation says allowed, so IAM is fine' and then spend an afternoon on the wrong layer.
aws iam simulate-principal-policy \--policy-source-arn arn:aws:iam::111122223333:role/AppRole \--action-names s3:GetObject \--resource-arns arn:aws:s3:::partner-data/report.pdf \--resource-policy file://bucket-policy.json \--permissions-boundary-policy file://boundary.json
| Simulation input | What it covers | What it misses if you omit it |
|---|---|---|
| --policy-source-arn | Identity policies attached to the principal | Everything else, by default |
| --resource-policy | The bucket, key, queue, or topic policy on the target | Resource-based Allows and Denys are invisible |
| --permissions-boundary-policy | The boundary attached to the principal | Boundary-caused denials are not reproduced |
| --context-entries | Condition keys such as aws:SourceVpce or aws:PrincipalOrgID | Any request that depends on a condition key fails or passes wrongly |
| (service control policies) | Evaluated when the principal's account is in an organization | Nothing for SCPs, but the error message will not name the SCP |
Do not fix an AccessDenied by widening the policy until it works. Replacing a scoped resource ARN with a wildcard, or adding NotAction-based Deny exceptions, routinely converts a five-minute debugging problem into an audit finding. Find the layer first, then grant the narrowest permission that resolves it and document why.
The Two Grants Cross-Account Access Always Needs
Within a single account, an Allow in either the identity policy or the resource policy is enough, because AWS takes the union of those two. Across accounts it is not a union. The calling principal needs an identity-based policy in its own account permitting the action, and the target resource needs a resource-based policy naming that principal as allowed. Satisfying only one of those produces AccessDenied with no further explanation, and it is the single most common cross-account failure.
A related trap is confusing the trust policy with the permissions policy on a role. The trust policy answers who may assume the role. The role's identity policy answers what the resulting session may do. A role can be assumable by your account and still be useless, because assume-role succeeds and every subsequent API call fails. If sts:AssumeRole works and nothing else does, you are looking at the wrong document.
{"Version": "2012-10-17","Statement": [{"Sid": "AllowPartnerReadOnly","Effect": "Allow","Principal": {"AWS": "arn:aws:iam::111122223333:role/AppRole"},"Action": ["s3:GetObject"],"Resource": "arn:aws:s3:::partner-data/report.pdf"}]}
That bucket policy alone is not sufficient for the cross-account call. AppRole also needs an identity policy in account 111122223333 granting s3:GetObject on the same object ARN. Bucket policies look like they should be authoritative; for cross-account access they are only half the answer.
The S3 and KMS Traps That Survive Every Other Check
S3 has two separate action namespaces and mixing them up is a classic. ListBucket is a bucket-level action and must be granted on arn:aws:s3:::bucket-name. GetObject and PutObject are object-level and must be granted on arn:aws:s3:::bucket-name/*. Grant GetObject on the bucket ARN and it does nothing; grant ListBucket on the object ARN and it does nothing either. Both mistakes look correct at a glance because the ARN is right in structure and wrong in level.
The trap that defeats even experienced engineers is SSE-KMS. An object encrypted with a customer-managed key requires a grant in the KMS key policy in addition to the S3 permission. S3 calls KMS on your behalf to decrypt, and KMS authorizes that call against the key policy, which is a separate authorization path from IAM. You can hold s3:GetObject on the exact object, have no Deny anywhere, and still get AccessDenied because the key policy grants decrypt to nobody but the account root. It is the denial that genuinely is not an IAM problem.
For any S3 AccessDenied on an object you know exists, check three things in order: is s3:ListBucket granted on the bucket ARN, is s3:GetObject granted on the object ARN, and is the object encrypted with a customer-managed KMS key whose key policy grants kms:Decrypt to your principal. That sequence resolves the large majority of S3 denials without touching CloudTrail.
A Workflow That Stops the Guessing
- Reproduce the failure with --debug on and capture the raw request ID from the response body.
- Match that request ID to the CloudTrail event, and read the errorCode and errorMessage fields rather than the event name alone.
- Identify the account boundary: is the caller and the target resource in the same account, or is this cross-account?
- If cross-account, confirm both the identity policy and the resource policy grant the action before looking anywhere else.
- If same-account, walk the evaluation order: explicit Deny, SCP, permission boundary, identity or resource Allow.
- Run simulate-principal-policy with --resource-policy and --permissions-boundary-policy supplied, and with --context-entries if the policy uses condition keys.
- If S3 with a customer-managed KMS key is involved, read the key policy separately; IAM results tell you nothing about it.
- ▪An explicit Deny anywhere outranks every Allow, so search for Deny statements before you search for missing grants.
- ▪Permission boundaries and service control policies can only restrict; a boundary that omits an action is the denial, not a bug.
- ▪The trust policy on a role governs assumption, not permissions; assume-role succeeding says nothing about what the session can do.
- ▪S3 returns 403 rather than 404 when ListBucket is missing, so an apparent missing-object error can be a permission error.
- ▪IAM Access Analyzer is the only reliable way to attribute a conditional denial to a specific statement and condition key.
The engineers who resolve AccessDenied fastest are not the ones who know IAM best; they are the ones who refuse to edit a policy before they know which layer rejected the request. Treat the error as the output of a six-stage evaluation and the debugging collapses from an afternoon of guessing to a handful of checks in a known order.