Identity and Access¶

Who can do what in an IOLake cluster: tenants and their admins, users and access keys, IAM policies, groups and roles, identity providers, and temporary credentials from an OIDC token (STS).

Everything here is reachable two ways: the admin API under /_admin/*, and the AWS IAM and STS APIs (aws iam, aws sts, boto3, Terraform's AWS provider). Both are signed with SigV4 like any S3 request and work on the same records: a user created with aws iam create-user is the user GET /_admin/users lists. The management SDKs wrap the admin API for Go and Python.

Scopes: the system and tenants¶

Every identity object lives in one scope.

The system scope belongs to the operator. Its admin (the credential meta prints on first boot) manages the whole cluster.
A tenant is a hard boundary. Its users, groups, roles, policies and buckets are its own. No policy can grant a tenant's identity access to another tenant's bucket: the check happens before any policy is read.

A tenant admin is a user of the tenant listed in its admin_users. A tenant admin manages everything in the tenant and nothing outside it.

# As the system admin
POST /_admin/tenants                    {"name": "acme", "display_name": "Acme", "enabled": true}
POST /_admin/users                      {"display_name": "acme-admin", "tenant": "acme"}
POST /_admin/tenants/acme/admins        {"user_id": "<user_id>"}
POST /_admin/users/<user_id>/access-keys {}

From then on the tenant admin works with its own key. Each call below acts in the caller's own tenant. The system admin can name a tenant with ?tenant= (or "tenant" in a body); a tenant admin can't name another tenant.

Disabling a tenant¶

Disable a tenant to cut off everyone in it at once, without deleting anything:

iolakectl tenant update acme --enabled false   # PUT /_admin/tenants/acme {"enabled": false}
iolakectl tenant update acme --enabled true    # and back
iolakectl tenant create acme --disabled        # set it up first, open it later

While a tenant is disabled, every identity that belongs to it is refused with 403 AccessDenied, the answer a suspended user's key gets, and the gateway logs which identity it refused and why. That covers:

its users' access keys, on every API: S3 (signed requests, presigned URLs and browser form uploads), the IAM and STS APIs, the admin API, and the Iceberg and Unity catalogs;
temporary credentials that act in the tenant: users' own sessions (GetSessionToken) and sessions of its roles, including a role assumed from outside the tenant;
console sessions: one already open is refused on its next request, and a new sign-in, with a key or through an identity provider, is refused;
new sessions for its roles: AssumeRole and AssumeRoleWithWebIdentity are refused;
Delta Sharing: a recipient's token reads none of the tenant's shares, and they are no longer listed to it. Links to data files that a recipient got from a query earlier stay valid until they expire (an hour by default).

The users of a tenant that has been deleted are refused the same way.

How fast: at once on the gateway that made the change, and within 15 seconds on every other gateway, the time a gateway keeps a credential.

Enabling the tenant restores everything. Keys, sessions and presigned URLs still within their lifetime work again; nothing has to be issued anew.

Disabling refuses identities, not the tenant's data. Its buckets and objects stay as they are, the system admin can still manage the tenant (create its users, keys and buckets while it is disabled), and access that names no identity of the tenant goes on: a bucket policy that grants anonymous access still does, where public access isn't blocked. Block public access for the tenant too (public access) if its buckets must go dark.

Quotas¶

A tenant can have quotas on bytes, objects and buckets, and a bucket can have quotas on bytes and objects. 0 means unlimited. Bytes and objects count every version stored, current and noncurrent; a tenant's usage is the sum of its buckets'.

iolakectl tenant create acme --quota-bytes 2T --quota-objects 10000000
iolakectl bucket set-quota photos --bytes 500G --objects 1000000
# or: PUT /_admin/buckets/{bucket}/quota  {"quota_bytes": N, "quota_objects": N}

A system admin sets tenant quotas. A bucket's quota can be set by a system admin or by its tenant's admin. The bucket quota is checked when a bucket is created.

A write that would pass a byte or object quota is refused with 403 QuotaExceeded: a PUT, a copy, a browser upload, an upload part, or a multipart completion. It is refused before any of its bytes are stored.

Usage comes from the usage report, which the OSDs feed. While any quota is set, each gateway refreshes the report every 5 s.
A delete makes room once the next report shows it.
Within one gateway, enforcement is exact, up to the writes already in flight when the limit is crossed.
With several gateways, each also allows what the others admitted within one report interval, so usage can pass a quota by that much. This is the same trade-off Ceph RGW and MinIO make. The design is in architecture/design/s3/quotas.md.

Refusals are counted in iolake_s3_quota_refusals_total{scope}.

Users and access keys¶


`GET /_admin/users`	users in scope
`POST /_admin/users` `{"display_name", "email"?, "tenant"?}`	create
`GET /_admin/users/{user_id}`	one user
`PUT /_admin/users/{user_id}` `{"status"?, "display_name"?, "email"?}`	update; `status` is `active` or `suspended`
`DELETE /_admin/users/{user_id}`	delete
`GET /_admin/users/{user_id}/access-keys`	the user's keys
`POST /_admin/users/{user_id}/access-keys` `{"scope"?, "operation"?}`	new key; a `scope` like `s3://bucket/prefix/` confines it
`PUT /_admin/access-keys/{id}` `{"status": "active" \\| "inactive"}`	turn a key off or back on
`DELETE /_admin/access-keys/{id}`	delete a key

Suspend rather than delete when you need to cut someone off now and decide later. A suspended user's keys are all refused, and deactivating a key refuses just that key. Nothing is deleted, and reactivating restores access. Gateways cache credentials for at most 15 seconds, so the change takes effect everywhere within that time.

A scoped key (one with a scope) can only reach that bucket or prefix, and is refused on the admin API entirely. A credential handed to a workload therefore can't mint itself a wider one.

User names are case-insensitive (Alice and alice are one user) and unique across the cluster, not only within a tenant. A user has a path (/, or /team/), which is part of its ARN: arn:objectio:iam::acme:user/team/alice. Renaming a user or moving it to another path changes its ARN; a tenant's admin_users entry that names it by ARN must then be updated (one naming its user id needn't).

Policies¶

IAM policies are AWS-format JSON documents: Effect, Action, Resource, Condition. They are attached to users, groups or roles. A bucket policy is attached to a bucket instead and set with S3's PUT /{bucket}?policy. The two are evaluated together:

An explicit Deny anywhere wins.
An Allow in an attached policy or the bucket policy grants access.
With no Allow, the bucket's owner is allowed and everyone else is denied.

No policy means owner-only, not open.


`GET /_admin/policies`	the tenant's own policies, plus the shared catalogue
`POST /_admin/policies` `{"name", "policy", "tenant"?, "shared"?}`	create
`GET /_admin/policies/{name}`	one policy
`PUT /_admin/policies/{name}` `{"policy"}`	replace the document
`DELETE /_admin/policies/{name}`	delete
`POST /_admin/policies/attach` `{"policy_name", "user_id" \\| "group_id" \\| "role_name"}`	attach
`POST /_admin/policies/detach`	detach, same body
`GET /_admin/policies/attached?user_id=\\|group_id=\\|role_name=`	what's attached

Policies are per tenant: acme's auditors and globex's auditors are different documents. The system admin can mark a system policy "shared": true so that every tenant can attach it but none can change it.

The built-in readonly, readwrite and writeonly policies are shared. consoleAdmin (admin:*) is the operator's alone.

AWS managed policies. AmazonS3FullAccess, AmazonS3ReadOnlyAccess, IAMFullAccess and IAMReadOnlyAccess are built in, the same in every cluster, and can't be changed or deleted. Attach one by its AWS ARN through the IAM API (arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess), or as aws:AmazonS3ReadOnlyAccess through the admin API.

Inline policies belong to one user, group or role, by name, and go when it goes. They are managed through the IAM API (PutUserPolicy, PutGroupPolicy, PutRolePolicy, …) and evaluated with the attached ones: a Deny in any of them wins.

s3:* grants S3 actions only. (It used to match every action, which would have let an S3 grant make IAM calls once those existed.) Action names are case-insensitive.

Deleting a role or a group, through either API, also deletes its inline policies and its attachments: a role created later under the same name starts with none. Deleting a user takes its inline policies, attachments and group memberships with it.

Attaching and detaching take effect immediately on the gateway that made the change, and within 15 seconds on the others.

Policies support every IAM condition operator, plus IfExists, ForAnyValue:/ForAllValues: and Null. A document using an unknown operator is refused when you save it rather than being silently ignored. Policies written for AWS work unchanged: arn:aws: ARNs and {"AWS": ...} principals mean the same as arn:obio: and {"OBIO": ...}. See condition keys.

Groups¶


`GET /_admin/groups`	groups in scope
`POST /_admin/groups` `{"group_name", "tenant"?}`	create
`GET /_admin/groups/{group_id}`	one group, with its members
`DELETE /_admin/groups/{group_id}`	delete
`POST /_admin/groups/{group_id}/members` `{"user_id"}`	add a member of the group's own tenant
`DELETE /_admin/groups/{group_id}/members/{user_id}`	remove

A user gets the policies attached to every group it belongs to.

Roles¶

A role is a set of policies that someone can take on for a while, rather than an identity of its own. Who may take it on is decided by its trust policy. A role is assumed with an OIDC token through STS, or by a user or another role's session with AssumeRole.


`GET /_admin/roles`	roles in scope
`POST /_admin/roles` `{"name", "trust_policy", "description"?, "max_session_seconds"?}`	create
`GET /_admin/roles/{name}`	the role and its attached policies
`PUT /_admin/roles/{name}` `{"trust_policy"?, "description"?, "max_session_seconds"?}`	update
`DELETE /_admin/roles/{name}`	delete

A role's ARN names its scope: arn:obio:iam::acme:role/ci for tenant acme, arn:obio:iam::objectio:role/oncall for the system.

Identity providers (OIDC)¶

The operator and each tenant can have their own OpenID Connect provider: Keycloak, Entra ID, Okta, Google, Dex, or any other compliant issuer.

The operator's provider signs in system administrators. It is set with the gateway's --oidc-* flags, or stored with "system_admin": true.
A tenant's provider signs in that tenant's users. It's stored under the name t-<tenant>, which a tenant admin can create for itself. That also binds it to the tenant.

PUT /_admin/config/identity/openid/t-acme
{"issuer_url": "https://keycloak.acme.example/realms/acme",
 "client_id": "iolake", "client_secret": "…",
 "claim_name": "groups", "scopes": "openid profile email"}

claim_name can be a dotted path, such as realm_access.roles for Keycloak realm roles.

Only the system admin can set system_admin, or edit or delete a provider that has it. Otherwise a tenant admin could point the operator's sign-in at their own IdP.

A tenant names its provider in its oidc_provider field. The system admin can set it to any stored provider when creating or updating the tenant (iolakectl tenant create|update --oidc-provider); a name that isn't stored under identity/openid/ is refused with 400. A provider that a tenant names can't be deleted: the delete answers 409, naming the tenants. Point them at another provider, or clear the field ("oidc_provider": ""), first. When a tenant admin deletes their own t-<tenant> provider, the tenant's binding to it is cleared with it.

A tenant admin can change, disable or delete only its own tenant's provider, and only while no other tenant uses it: a provider that another tenant names in its oidc_provider, or whose t-<tenant> name is another tenant's, signs users in to that tenant too, so only the system admin can change it. A tenant's users who aren't its admins can't change it at all.

A provider stored with "enabled": false signs no one in, and no one signs in to the console of a disabled tenant, through a provider or with a key.

Signing in to the console¶

Who a sign-in is. The first sign-in of an identity creates a user for it, bound to the token's issuer (iss) and subject (sub). Every later sign-in with that issuer and subject is that user, whatever name the token carries by then. The user is named after the token's preferred_username, else its email, else its sub; the name is only a label, and a sign-in is never matched to an existing user by name. If another user already has the name, the sign-in is refused: rename or delete that user, or have the provider send another name. A token must carry iss, aud and sub, and its issuer must be the provider's.

Which tenant. A tenant's provider signs in to its tenant only: the tenant t-<tenant> names, or a tenant whose oidc_provider names the provider. A provider that several tenants name signs in to the one whose sign-in page the user started from, and to no other. An identity stays in the tenant it was first created in.

Who administers. A sign-in through the operator's provider is a system administrator on the console and the admin API. That holds for the session only: an access key the user creates for themselves is their own user's, not the system admin's. A user the system admin creates in the system scope is not an administrator either; only the system admin's own credentials and the operator's provider are. A tenant's users administer their tenant when they are in its admin_users, which a provider fills on sign-in from a role claim when it sets tenant_admin_role.

The bindings are format level 9: on a cluster upgraded from an earlier level, console sign-in through a provider is refused until iolakectl upgrade finalize. Access-key sign-in works throughout. A console session opened by a gateway of the earlier release isn't accepted by an upgraded one: sign in again.

Temporary credentials from an OIDC token (STS)¶

AssumeRoleWithWebIdentity trades an OIDC token for a role's temporary S3 keys. It's the AWS STS call, so AWS SDKs and tools speak it as is. Point them at the gateway's S3 endpoint:

import boto3

sts = boto3.client("sts", endpoint_url="https://s3.example.com", region_name="us-east-1")
r = sts.assume_role_with_web_identity(
    RoleArn="arn:obio:iam::acme:role/ci",
    RoleSessionName="build-42",
    WebIdentityToken=open("/var/run/secrets/token").read(),
    DurationSeconds=3600,
)
c = r["Credentials"]
s3 = boto3.client("s3", endpoint_url="https://s3.example.com",
                  aws_access_key_id=c["AccessKeyId"],
                  aws_secret_access_key=c["SecretAccessKey"],
                  aws_session_token=c["SessionToken"])

Which provider can vouch for which role:

role	accepted tokens
system role (`arn:obio:iam::objectio:role/…`)	the operator's providers only
tenant role (`arn:obio:iam::acme:role/…`)	that tenant's provider only. For a multi-tenant provider such as Entra `organizations`, only tokens from the upstream organisation registered to the tenant

A token from the wrong provider is refused with InvalidIdentityToken, however its claims read.

The trust policy then decides. Claims are condition keys named after the issuer (without https://): <issuer>:sub, <issuer>:aud, <issuer>:email, <issuer>:groups and any other claim. As in AWS, a trust policy has no Resource.

{"Version": "2012-10-17", "Statement": [{
  "Effect": "Allow",
  "Principal": {"Federated": "https://keycloak.acme.example/realms/acme"},
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {"keycloak.acme.example/realms/acme:aud": "iolake"},
    "ForAnyValue:StringEquals": {"keycloak.acme.example/realms/acme:groups": "ci"}
  }
}]}

What the credentials can do: what the role's attached policies allow, inside the role's tenant. Detaching a policy affects sessions already issued. A system role is not a system admin.

How long they last: DurationSeconds is between 900 and the role's max_session_seconds. The default is 3600 and the maximum 12 hours.

Errors: AccessDenied (trust policy, or no such role: the two are answered the same, so role names can't be probed), InvalidIdentityToken, ExpiredTokenException, ValidationError.

The AWS IAM and STS APIs¶

The gateway answers the AWS IAM and STS query APIs on its S3 endpoint: POST / with a form body (or GET /?Action=…), signed with SigV4 for the service iam or sts in any region. Point the client at the S3 endpoint:

aws --endpoint-url https://s3.example.com iam create-user --user-name alice --path /team/
aws --endpoint-url https://s3.example.com iam create-access-key --user-name alice
aws --endpoint-url https://s3.example.com iam attach-user-policy --user-name alice \
    --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
aws --endpoint-url https://s3.example.com sts get-caller-identity

iam = boto3.client("iam", endpoint_url="https://s3.example.com", region_name="us-east-1")
sts = boto3.client("sts", endpoint_url="https://s3.example.com", region_name="us-east-1")

Accounts¶

A tenant is an IAM account; its account id is the tenant's name. The system scope is the account with the empty id. A call acts in the caller's own account: a user of acme creates users in acme, and nothing of another tenant exists for it (NoSuchEntity).

Who may make which call:

Caller	May
the system admin	any call, in the system account
a tenant admin	any call, in its tenant: it is the account's root
any other user, or a role's session	what its own policies allow (`iam:CreateUser` on `arn:aws:iam::acme:user/team/*`, …)
a key scoped to a bucket, the data-lake APIs' vended credentials	none

GetUser (with no user name) and GetCallerIdentity name the system admin and tenant admins arn:aws:iam::<account>:root.

ARNs¶

The IAM API shows ARNs as AWS does. Policies may use either spelling.

	IAM API	stored (admin API)
user	`arn:aws:iam::acme:user/team/alice`	`arn:objectio:iam::acme:user/team/alice`
system user	`arn:aws:iam:::user/ops`	`arn:objectio:iam::user/ops`
group	`arn:aws:iam::acme:group/devs`	`arn:obio:iam::acme:group/devs`
role	`arn:aws:iam::acme:role/apps/ci`	`arn:obio:iam::acme:role/apps/ci`
policy	`arn:aws:iam::acme:policy/readers`	`acme/readers`
a shared system policy	`arn:aws:iam:::policy/readonly`	`readonly`
AWS managed policy	`arn:aws:iam::aws:policy/IAMReadOnlyAccess`	`aws:IAMReadOnlyAccess`
a role's session	`arn:aws:sts::acme:assumed-role/ci/build-42`	`arn:obio:sts::acme:assumed-role/ci/build-42`

In policies, arn:aws:, arn:obio: and arn:objectio: ARNs are the same ARN, and an empty account and objectio both name the system scope.

Actions¶

Area	Actions
Users	`CreateUser`, `GetUser`, `ListUsers`, `UpdateUser` (rename, move), `DeleteUser`
Access keys	`CreateAccessKey`, `ListAccessKeys`, `UpdateAccessKey`, `DeleteAccessKey`
Groups	`CreateGroup`, `GetGroup`, `ListGroups`, `UpdateGroup` (rename, move), `DeleteGroup`, `AddUserToGroup`, `RemoveUserFromGroup`, `ListGroupsForUser`
Managed policies	`CreatePolicy`, `GetPolicy`, `GetPolicyVersion`, `ListPolicyVersions`, `ListPolicies`, `DeletePolicy`
Attachments	`Attach`/`Detach`/`ListAttached` `UserPolicies`, `GroupPolicies`, `RolePolicies`
Inline policies	`Put`/`Get`/`List`/`Delete` `UserPolicy`, `GroupPolicy`, `RolePolicy`
Roles	`CreateRole`, `GetRole`, `ListRoles`, `UpdateRole`, `UpdateAssumeRolePolicy`, `DeleteRole`
Account	`GetAccountSummary`, `ListOpenIDConnectProviders`
STS	`GetCallerIdentity`, `GetSessionToken`, `AssumeRole` (and `AssumeRoleWithWebIdentity`, unsigned)

Lists page with Marker and MaxItems and filter with PathPrefix; they sort by name, ignoring case.

As in IAM: names are case-insensitive; deleting a user with access keys or policies, a group with members or policies, or a role with policies is a DeleteConflict; inline policies are at most 2048 (user), 5120 (group) or 10240 (role) characters, managed ones 6144, whitespace aside.

STS¶

AssumeRole: the role's trust policy decides, with the caller as principal (its ARN, and for a user its id too: …:user/<user_id>). Within a tenant that is enough. A caller from another tenant must be named by the trust policy (a grant to * stays in the role's tenant) and its own policies must allow sts:AssumeRole on the role, as IAM's cross-account rule has it; a trust policy naming an account's root (arn:aws:iam::acme:root) lets in that account's users whose own policies allow it. The session acts as the role: its policies (attached and inline), its tenant. 15 minutes to the role's MaxSessionDuration (default 1 hour); a role's session assuming another gets at most 1 hour. Session policies (Policy, PolicyArns) are refused.
GetSessionToken: a user's own access key gets temporary keys that act as the user (its policies, groups, tenant), for 15 minutes to 36 hours (default 12). Requests with them carry sts:authentication = true for policy conditions. Suspending the user stops them within 15 seconds.
GetCallerIdentity: who the credentials are: user, root, or <RoleId>:<session> for a role's session.

The credentials work against S3 at once, on every gateway.

Not supported¶

OIDC provider management (CreateOpenIDConnectProvider, …): identity providers are configured through the admin API; ListOpenIDConnectProviders lists those the account trusts.
Policy versions beyond one: a policy has one version, v1 (CreatePolicyVersion isn't supported; change a policy through the admin API's PUT /_admin/policies/{name}).
Login profiles, MFA, signing certificates, SSH keys, service-specific credentials, tags, permission boundaries, instance profiles, GetFederationToken, SimulatePrincipalPolicy.
IAM quotas: none are enforced. GetAccountSummary reports the counts, and AWS's default quotas for clients that read them.
DeletePolicy detaches the policy everywhere rather than refusing while it is attached.
ListBuckets is never refused for lack of a grant: it lists the buckets the caller owns (AWS refuses a user no policy allows s3:ListAllMyBuckets). An explicit Deny of s3:ListAllMyBuckets refuses it.

Upgrading¶

Paths, descriptions of policies, group renames and inline policies work on every release. A role or policy created without an id (on 0.1.0-rc.1 or rc.2, before the cluster's first OSD registered) gets one derived from its ARN and creation time.

Session and token signing¶

Temporary credentials, console sessions and listing continuation tokens are signed with keys derived from a cluster secret. Meta creates the secret once and replicates it through Raft, so every gateway shares it: a session issued by one gateway is honoured by all of them, and nothing about it can be forged from outside. The secret is never exposed by any API.

ACLs¶

Buckets are bucket owner enforced, the S3 default since 2023. The bucket owner owns every object, ACLs grant nothing, and access is decided by policies only. ACL reads answer with the owner's FULL_CONTROL. An ACL write that says only that is accepted; any other ACL write is refused with AccessControlListNotSupported. No ACL can make data public: see Public access for how that's done deliberately.