S3 compatibility: ceph s3-tests¶
IOLake is tested against ceph/s3-tests, the S3 compatibility suite RGW is built against: 1053 tests. Each run is in a dated folder here, with every test's outcome and, for each failure, why it fails.
Latest: 2026-11-05 (0.1.0-rc.4)¶
Passes 681 of 687 applicable tests on Linux x86_64, and 679 on Linux arm64. The 6 that don't pass on x86_64 are the gaps below, test for test as on 0.1.0-rc.3: no test passes or fails that didn't before. On arm64, run on 0.1.0-rc.1, the same 6 gaps, and 2 tests that fail only when run in parallel (below); none is a bug.
| Platform | IOLake | Summary | Every test |
|---|---|---|---|
| Linux x86_64 (NVMe) | 0.1.0-rc.4 | summary | results.csv |
| Linux arm64 (Ubuntu 24.04 VM on Apple silicon) | 0.1.0-rc.1 | summary | results.csv |
s3-tests 5522d1c, now 1057 tests; a debug iolake-aio (6 OSDs, 4+2
erasure coding, auth on), fresh for the run. arm64 ran on 2026-10-18,
on the tagged 0.1.0-rc.1; its 1057 tests fall into the same groups below,
test for test, apart from the three named at the end.
Applicable, and not¶
A test is applicable when it isn't skipped and doesn't fail for a reason that isn't IOLake's to fix. Of the 1057:
| Tests | What | |
|---|---|---|
| Applicable | 687 | 681 pass (x86_64) |
| Skipped by the suite | 92 | S3 tests the suite itself skips here |
| Not implemented, by design | 189 | below |
| Test environment | 41 | 27 need a web-identity provider for STS; 6 expect RGW's tenant-less IAM ARNs; the rest are the suite's own assumptions (botocore re-signing a header the test removed, a header on the wrong client, a leftover bucket, virtual-host DNS) |
| IOLake does what AWS does | 48 | the test expects RGW's behaviour instead (owner display names, how bucket logs name a copy or a batch delete, <And> in lifecycle filters, NoSuchBucketPolicy, S3 Select's AWS output, …) |
Not implemented, by design:
| Tests | What | Plan |
|---|---|---|
| 107 | ACLs (grants, x-amz-grant-*, public-read ACLs, ownership controls) |
Never: buckets are BucketOwnerEnforced, AWS's default since 2023; bucket policies instead |
| 46 | SigV2 (headers, POST policy, presigned URLs) | Never: AWS deprecated it; SigV4 only |
| 13 | Access across tenants (bucket policies, role assumption, a log target in another tenant) | Never: a tenant is a hard boundary |
| 9 | SNS (topics, bucket notifications) | Not planned |
| 8 | RGW extensions (unordered listing, tenant:bucket, usage API, X-RGW-* headers) |
Never: not S3 |
| 4 | ListBuckets refused to a user no policy allows | ListBuckets lists what the caller owns; only an explicit Deny refuses it |
| 1 | Torrent | Never: AWS dropped it |
| 1 | IAM CreateRole as the test calls it |
Gaps (6):
- An account root (tenant admin) isn't exempt from its own bucket's policy for the bucket-policy calls, as AWS's root is (1).
- IAM OpenID Connect provider management (5): providers are configured
through the admin API, and
ListOpenIDConnectProviderslists them.
Not a gap: on 2026-10-15, test_s3select::test_cast_expressions
passed, but its teardown failed. It lists buckets as the suite's second
user while another test, running at the same time, denies that user
ListBuckets for a few seconds (test_deny_bucket_actions_in_user_policy).
Run alone, it passes with its teardown, as it did in the runs since.
On x86_64 for 0.1.0-rc.4, a first run failed
test_sts::test_get_caller_identity_non_root_user with
EntityAlreadyExists: it and test_get_caller_identity_after_assume_role
both create the IAM user s3-tests-MyUser, and on parallel workers the two
collide. Alone or with one worker it passes; the published run is the
second, where it passed.
On arm64 that test passed, and two others failed only in the parallel run; run alone, each passed three times out of three:
test_headers::test_object_create_bad_md5_bad: the same interference, its ListBuckets denied by the other test's user policy.test_s3::test_bucket_logging_single_prefix: found two log objects where it expects one. Logs roll every 3 seconds for the suite and the test waits 5.5; with the VM busy running four workers, a second roll landed inside its wait.
Running it¶
From the iolake repo, on Linux (on a Mac, inside a Linux VM or container: native macOS syncs are far slower, and its timings mean nothing):
scripts/s3-tests/run.sh # about 10 minutes
scripts/s3-tests/run.sh --previous <last run>/results.csv # lists what changed
See scripts/s3-tests/README.md there. To publish a run, copy its
report/ folder here as <date>/<platform>/ and update the numbers above.
Failures are labelled from scripts/s3-tests/known-failures.csv; a new
failure shows up as uncategorized until it is fixed or given a reason
there.
Runs¶
| Date | IOLake | Applicable passed | Passed | Failed | Skipped |
|---|---|---|---|---|---|
| 2026-11-05 | 0.1.0-rc.4, Linux x86_64 | 681 of 687 | 681 | 280 (+4 errors) | 92 |
| 2026-10-23 | 0.1.0-rc.3, Linux x86_64 | 681 of 687 | 681 | 280 (+4 errors) | 92 |
| 2026-10-21 | 0.1.0-rc.2, Linux x86_64 | 681 of 687 | 681 | 280 (+4 errors) | 92 |
| 2026-10-18 | 0.1.0-rc.1, Linux arm64 | 679 of 687 | 679 | 281 (+5 errors) | 92 |
| 2026-10-15 | 0.1.0 release candidate, Linux x86_64 | 680 of 687 | 680 | 280 (+5 errors) | 92 |
| 2026-10-03 (x86_64, arm64) | development build, Linux x86_64 and arm64 | 591 of 591 | 591 | 300 | 162 |
| 2026-10-02 | development build, macOS | 586 of 593 | 586 | 305 | 162 |
| 2026-10-02 (earlier) | development build (before run 2's fixes), macOS | 580 | 311 | 162 |