Skip to content

ceph s3-tests against IOLake, run 2

  • Date: 2026-10-02
  • IOLake: a development build of 2026-10-02 with the fixes from run 1, debug objectio-aio
  • s3-tests: 5522d1c351f75bc00ae0f64f742f3f095f5939d9 (Merge pull request #669, 2026-05-27)
  • Server: objectio-aio --osds 6 --ec-k 4 --ec-m 2 --auth --lifecycle-interval-secs 2 --lifecycle-day-secs 10 on 127.0.0.1:19800, fresh data dir
  • Invocation (same as run 1): pytest -n 4 s3tests/functional, no marker exclusions, all 1053 tests collected
  • Users: tenant s3t (main, alt, iam), tenant testx (tenant), tenants iamr/iamalt (iam root users); cluster default public-access block for new buckets turned off; KMS keys testkey-1, testkey-2. The system admin is not used as any test user.

Totals

Run 1 (previous) Run 2 (this) Change
Passed 580 586 +6
Failed 311 305 -6
Skipped 162 162 0
Error 0 0 0
Total 1053 1053

All 305 failures were rerun serially (pytest --lf, no xdist): all 305 failed again. No flakes. Nothing passed in run 1 and failed in run 2.

Failures by category

Category Count
not-implemented-by-design 235
test-environment 35
aws-compatible-already 28
gap 4
bug 3
Total 305

not-implemented-by-design (235): ACLs other than private / bucket is BucketOwnerEnforced (86, plus 8 more ACL or ownership rows: 94 in all); bucket logging (39); IAM API (27); SigV2 header auth (21); POST object signed with SigV2 (21); S3 Select (12); STS AssumeRole / GetSessionToken / GetCallerIdentity, which need the IAM API (9); SigV2 presigned URLs (4); RGW extensions: allow-unordered listing (2), tenant:bucket naming (2), usage / x-rgw-object-count headers (3); torrent (1).

test-environment (35): no [webidentity] section / IdP for AssumeRoleWithWebIdentity (27); header-removal tests where botocore signs or re-adds the header after the test removed it (5, all fails_on_rgw); test_list_buckets_paginated assumes the main user owns no buckets but another test left testbuck2 (1); test_bucket_policy_put_obj_request_obj_tag adds the tagging header to the wrong client (1); s3control needs <account-id>.127.0.0.1 virtual-host DNS (1).

aws-compatible-already (28): Owner DisplayName is the canonical id, test wants the RGW display name (6); lifecycle filters with several conditions outside <And> rejected, as AWS does (6, all fails_on_aws); GetBucketPolicyStatus on a bucket with no policy is NoSuchBucketPolicy (3); BucketOwnerEnforced means a copied object belongs to the bucket owner (2); malformed checksum value gives InvalidRequest, RGW says BadDigest (2); and one each: anonymous ListBuckets 403, us-east-1 LocationConstraint is null, duplicate part number in CompleteMultipartUpload rejected, ListObjects v1 percent-encodes Prefix under encoding-type=url, owner request not denied by an Allow-only policy, non-ASCII key read 404, non-ASCII metadata returned RFC 2047 encoded, GetBucketOwnershipControls always answers BucketOwnerEnforced, HEAD of missing key without GetObject is 403.

Borderline calls, in case someone wants to revisit them: - test_bucket_list_prefix_unreadable: AWS documents Prefix as encoded under encoding-type=url, but botocore does not decode the v1 Prefix and RGW sends it raw. Not checked against real AWS. - test_multipart_resend_first_finishes_last: the parts list has part 1 twice. AWS would reject it too, with InvalidPartOrder; IOLake's error is EntityTooSmall. The error code is wrong but the outcome is the same. - DisplayName tests: AWS stopped returning DisplayName in late 2025. IOLake fills DisplayName with the user id (bin/objectio-gateway/src/s3.rs:4459) rather than leaving it out.

By area

Area Passed Failed Skipped Main failure reasons (count)
encryption 146 0 0
logging 0 39 74 bucket logging (38); bucket logging + IAM CreateRole (1)
iam 0 27 63 IAM API (27)
listing 78 7 1 RGW allow-unordered listing extension (2); ACLs (2); anonymous ListBuckets is 403 as on AWS (1)
acl/ownership 8 56 0 ACLs (41); Owner DisplayName is the canonical user id (5); first call on bucket without policy -> NoSuchBucketPolicy, as AWS (2)
aws2/headers 21 27 0 SigV2 (21); botocore signs/re-adds the header after the test removes it (5); ACLs beyond canned private (1)
lifecycle 26 9 12 Prefix+Tag without rejected, as AWS (4); no x-amz-expiration response header (3); several filter conditions without rejected, as AWS (2)
bucket 33 9 0 ACLs (4); RGW x-rgw-object-count header (1); us-east-1 returns null LocationConstraint as AWS does (1)
policy/public-access 22 18 2 ACLs (10); RGW tenant:bucket naming (2); BucketOwnerEnforced: copied object belongs to bucket owner, alt has no (2)
post_object 12 28 0 POST object signed with SigV2 (21); ACLs (7)
object_lock 39 0 0
versioning 36 1 1 ListObjectVersions entries carry no Owner element (1)
multipart 34 2 0 completes with part 1 listed twice (1); ACLs (1)
sts 0 36 0 no [webidentity] section / IdP configured (27); STS AssumeRole/GetSessionToken/GetCallerIdentity need the IAM API (9)
select 22 12 0 S3 Select (12)
object 24 8 0 ACLs (6); Content-Encoding aws-chunked with a plain (1); non-ASCII metadata returned RFC 2047 encoded as S3 does (1)
ranged/get 16 6 1 ACLs (4); non-ASCII key read returns 404 NoSuchKey (1); torrent (1)
conditional 23 0 0
copy 16 1 0 ACLs (1)
tagging 16 0 0
other 9 1 5 ACLs (1)
cors 1 13 0 ACLs (9); SigV2 presigned URLs (4)
checksum 4 2 0 malformed checksum value 'bad' -> InvalidRequest as AWS (2)
storage_class 0 0 3
usage 0 2 0 RGW usage extension (1); RGW X-RGW-* usage headers (1)
s3control 0 1 0 needs .host virtual-host DNS (1)

Areas come from the test file (iam, sts, s3select, s3control, headers) or, for test_s3.py, from keywords in the test name (first match wins), as in run 1's cat.py. Skips: 74 "ceph extension to bucket logging not supported at client", 70 "[iam root] user does not belong to an account", 11 "[s3 cloud] section missing", 6 storage-class count / cloud-tier skips, 1 parameter-set skip. The skips are the same as in run 1.

Bugs (3)

  1. test_object_content_encoding_aws_chunked: PutObject with Content-Encoding: gzip, aws-chunked and an ordinary body (normal SigV4 payload hash, not STREAMING-*) fails with 400 IncompleteBody "body ended before the terminal chunk". is_s3_chunked treats any aws-chunked in Content-Encoding as chunked framing, even when x-amz-content-sha256 is not STREAMING-*. Location: bin/objectio-gateway/src/chunked_decode.rs:23-31. The gzip and deflate, gzip steps of the test now pass (both failed in run 1).
  2. test_create_bucket_bucket_owner_preferred: CreateBucket with x-amz-object-ownership: BucketOwnerPreferred succeeds, and GetBucketOwnershipControls then answers BucketOwnerEnforced. The header is ignored without any error, while PutBucketOwnershipControls rejects the same value with InvalidRequest. create_bucket never reads the header: bin/objectio-gateway/src/s3.rs:2763 (compare put_ownership_controls, s3.rs:4586).
  3. test_create_bucket_object_writer: same as 2, with ObjectWriter.

Gaps (4)

  1. test_bucket_list_return_data_versioning: ListObjectVersions <Version> entries have no <Owner>. AWS includes Owner (ID) there. ObjectVersionXml, bin/objectio-gateway/src/s3.rs:11693 (and DeleteMarkerXml at 11711).
  2. test_lifecycle_expiration_header_put: no x-amz-expiration header on PutObject when a lifecycle Expiration rule applies. Nothing in the gateway emits this header (no match for x-amz-expiration in bin/ or crates/). Rule evaluation is in bin/objectio-gateway/src/lifecycle.rs.
  3. test_lifecycle_expiration_header_head: same, on HeadObject.
  4. test_lifecycle_expiration_header_tags_head: same, for a tag-filtered rule on HeadObject.

Previously failing, now passing (6)

Test Fix
test_headers.py::test_object_create_date_and_amz_date Date header signing
test_headers.py::test_object_create_amz_date_and_no_date Date header signing
test_s3.py::test_bucket_create_exists_nonowner CreateBucket 409 BucketAlreadyExists
test_s3.py::test_delete_marker_suspended null delete marker with versioning suspended
test_s3.py::test_lifecycle_deletemarker_expiration_with_days_tag lifecycle checks the delete marker's age
test_s3.py::test_set_bucket_tagging bucket tagging

Fixes that changed a test's result without making it pass: - Non-ASCII metadata: test_object_set_get_unicode_metadata used to fail with AccessDenied "invalid header value" on PUT. It now stores the value and returns it RFC 2047 encoded, which is what S3 does. The test, marked fails_on_rgw, wants the raw value. - gzip, aws-chunked: test_object_content_encoding_aws_chunked used to fail at its first step (gzip not returned). The first two steps now pass and it fails at the third (bug 1 above). - DELETE on unhandled sub-resources: test_bucket_create_delete_bucket_ownership used to fail with NoSuchBucket, because DELETE ?ownershipControls deleted the bucket. Now the DELETE is refused (InvalidRequest, ownership cannot be removed) and the bucket survives. It still fails because IOLake supports only BucketOwnerEnforced, by design. - The aws-chunked IncompleteBody handling and the DELETE-in-own-task change have no dedicated s3-tests test.

Files

  • results.csv: one row per test (test_id, file, area, outcome, category, note)
  • junit-run2.xml: the parallel full run
  • junit-run2-serial.xml: serial rerun of the 305 failures
  • The run was made by hand; the same setup is now scripts/s3-tests/run.sh in the iolake repo.

Fixed since this run

  • Bug 1 (gzip, aws-chunked over a plain body; test_object_content_encoding_aws_chunked passes). Bugs 2 and 3: CreateBucket now refuses ownership other than BucketOwnerEnforced, as PutBucketOwnershipControls does, so those two tests now fail by design rather than by a silent mismatch.