Gateway-side grep — searching objects without downloading

When an agent (or a fleet of 100 clients) needs to search inside object contents, the naive approach is GET the object and grep client-side. For a 10 GB log file and 100 agents that's 1 TB of egress for one search — unacceptable.

IOLake ships two grep endpoints that run the scan on the gateway and stream only matching records (with their byte offsets) back to the caller. Built for log / text / markdown / JSONL workloads where line-level regex is the right tool; use the Iceberg catalog for structured tabular queries.

Single-object grep

POST /{bucket}/{key}?grep
Content-Type: application/json
{
  "pattern":          "ERROR.*timeout",
  "literal":          false,
  "case_insensitive": false,
  "max_matches":      100,
  "content_max_bytes": 2048,
  "invert":           false,
  "engine":           "regex"   // optional — "regex" | "pcre2" | "hyperscan"
}

Response is application/x-ndjson — one JSON event per newline:

{"type":"start","file_size":271,"etag":"\"abc123\"","pattern":"ERROR.*timeout"}
{"type":"match","line_number":3,"offset":80,"line_length":57,"match_offset":100,"match_length":13,"content":"2026-04-22 10:00:03 ERROR timeout connecting to host=db1"}
{"type":"match","line_number":5,"offset":179,"line_length":57,"match_offset":199,"match_length":13,"content":"2026-04-22 10:00:05 ERROR timeout connecting to host=db2"}
{"type":"end","matches":2,"bytes_scanned":271,"truncated":false,"elapsed_ms":1}

Every match frame gives the agent everything needed to find or fetch the source context:

Field	Purpose
line_number	1-indexed line in the source object
offset	Byte offset of the line start in the source file — use with Range: bytes=offset-
line_length	Bytes the line occupies (including the trailing newline if present)
match_offset	Byte offset of the matched substring itself
match_length	Length of the matched substring in bytes
content	The matched line, truncated to content_max_bytes (default 8 KiB)
truncated	true if the content field was clipped — agent can use offset + line_length to Range-fetch the full line

Multi-object (prefix) grep

POST /{bucket}?grep
Content-Type: application/json
{
  "pattern":                "panic",
  "prefix":                 "2026-04/",
  "continuation_token":     "",        // paginate from a prior scan's next_continuation_token
  "max_keys":               100,
  "max_matches":            500,
  "max_matches_per_object": 20,
  "literal":                false,
  "case_insensitive":       false,
  "content_max_bytes":      2048,
  "invert":                 false
}

Response is the same NDJSON shape, with per-key brackets:

{"type":"start","file_size":0,"etag":"","pattern":"panic"}
{"type":"object","bucket":"logs","key":"2026-04/app-20.log","file_size":151,"etag":"\"c19ddee6…\""}
{"type":"match","bucket":"logs","key":"2026-04/app-20.log","line_number":3,"offset":93,...}
{"type":"match","bucket":"logs","key":"2026-04/app-20.log","line_number":7,"offset":410,...}
{"type":"object_end","bucket":"logs","key":"2026-04/app-20.log","matches_in_object":2,"per_object_truncated":false}
{"type":"object","bucket":"logs","key":"2026-04/app-21.log","file_size":151,"etag":"..."}
...
{"type":"end","matches":6,"bytes_scanned":453,"truncated":false,"elapsed_ms":5,"objects_scanned":3}

Every match now carries bucket and key so agents can cross-reference.

Agent pattern: grep then Range-fetch

The canonical LLM workflow: small matches go straight into context; larger fetches happen only when the agent decides they're needed.

# Step 1 — grep
hits = tool.grep_object(
    bucket="logs", key="app.log",
    pattern="ERROR.*user=alice", max_matches=50)

# Step 2 — optional: fetch surrounding context around interesting match
m = hits[3]
chunk = tool.get_object(
    "logs", "app.log",
    range_header=f"bytes={m['offset']-4096}-"
                 f"{m['offset']+m['line_length']+4096}")

# Or: fetch the whole file if small
if hits.file_size < 1_000_000:
    full = tool.get_object("logs", "app.log")

MCP tool stub

Drop into any agent runtime that understands MCP-style tool schemas (Claude Code, OpenAI function calling, Anthropic MCP):

{
  "name": "grep_object",
  "description": "Search inside an object using a regex. Returns matching lines with byte offsets so the agent can fetch surrounding context via Range reads.",
  "input_schema": {
    "type": "object",
    "required": ["bucket", "key", "pattern"],
    "properties": {
      "bucket":           {"type": "string"},
      "key":              {"type": "string"},
      "pattern":          {"type": "string", "description": "Regex or literal. Set literal=true to disable regex interpretation."},
      "literal":          {"type": "boolean", "default": false},
      "case_insensitive": {"type": "boolean", "default": false},
      "max_matches":      {"type": "integer", "default": 100},
      "content_max_bytes":{"type": "integer", "default": 2048},
      "invert":           {"type": "boolean", "default": false}
    }
  }
}

And for prefix grep:

{
  "name": "grep_prefix",
  "description": "Search every object under a key prefix. Returns matches tagged by bucket+key so the agent knows which file each hit came from.",
  "input_schema": {
    "type": "object",
    "required": ["bucket", "pattern"],
    "properties": {
      "bucket":                  {"type": "string"},
      "pattern":                 {"type": "string"},
      "prefix":                  {"type": "string", "default": ""},
      "continuation_token":      {"type": "string", "default": "", "description": "Pass back the prior response's next_continuation_token to resume a paginated scan."},
      "max_keys":                {"type": "integer", "default": 100},
      "max_matches":             {"type": "integer", "default": 500},
      "max_matches_per_object":  {"type": "integer", "default": 20},
      "literal":                 {"type": "boolean", "default": false},
      "case_insensitive":        {"type": "boolean", "default": false}
    }
  }
}

Both tools inherit normal S3 auth — the agent must sign with SigV4 credentials that have s3:GetObject on the target keys, same as any other read.

Pattern engines

By default the scanner uses the pure-Rust regex crate — RE2-based, linear-time, no catastrophic backtracking. It handles nearly every pattern an agent will throw at it. When you need features the default can't express, pass "engine": "<name>" in the request body:

Engine	Compiled into	Adds
regex (default)	always	—
pcre2	Linux release binaries (--features grep-pcre2)	backrefs (\1), lookaround ((?=foo), (?<!bar)), atomic groups, recursion, named captures
hyperscan	Linux x86_64 release binaries (--features grep-hyperscan)	Intel SIMD-accelerated matching for high-throughput scanning over large streams

Selecting an engine that wasn't compiled into your gateway binary returns 400 BadRequest with a clear message:

HTTP 400: pattern compile failed: engine 'pcre2' is not compiled into
this gateway; rebuild with `--features pcre2` or select another engine

macOS release binaries ship with regex only. Linux arm64 ships with regex + pcre2. Linux amd64 ships with all three.

Example — duplicate-word detection (needs a backref, so regex can't do it; pcre2 can):

body = {"pattern": r"(\w+) \1", "engine": "pcre2"}
# → matches "hello hello", "foo foo", ...

What this is and isn't

Good fit:

• Plain text (.txt) logs and exports
• Markdown documents (.md)
• JSONL (one JSON object per line)
• Source code files where regex grep is already the right tool
• Any workload where "find matching lines without downloading the full file" is the ask

Not the right tool — use these instead:

Workload	Use
SQL over a table / filter a column	Iceberg REST Catalog + DuckDB / Spark / Trino
Column pruning on Parquet without a catalog	DuckDB over S3 with httpfs, client-side
Full-text search with relevance / BM25	Quickwit (S3-backed index, lightweight) or OpenSearch alongside the gateway
Code search with symbol awareness	Sourcegraph / Zoekt mirror
Pretty-printed JSON with structural path ($.users[?(@.name=="alice")])	Pipe the raw object through jq client-side; grep can't give you structural semantics

Limits and follow-ups

• Object body is collected into memory before scanning — fine up to a few GiB, hard cap at 16 GiB. Streaming off the EC read path is a planned follow-up that lifts the cap entirely.
• . in regex doesn't cross line boundaries — line-based grep by design; multiline patterns aren't supported in v1.
• Pretty-printed or single-line JSON has weaker match locality because there's no line structure. Use content_max_bytes and Range-fetch the offset to recover context.
• Per-request timeout follows the gateway's HTTP timeout (60s default). Patterns that require scanning very large prefixes should page by adjusting max_keys across requests.
• Pagination is supported via continuation tokens. When a prefix scan hits max_keys, the end frame includes next_continuation_token. Pass it back in the next request's continuation_token to resume. A response without next_continuation_token signals that the prefix is exhausted.

ct = ""
while True:
    resp = grep_prefix(bucket="logs", pattern="panic",
                       prefix="2026-04/", max_keys=100,
                       continuation_token=ct)
    # ... process matches ...
    ct = resp.end.get("next_continuation_token") or ""
    if not ct:
        break

Example — from the command line

curl using SigV4 via awscurl / aws s3api helpers. The S3 API takes the body's hash from x-amz-content-sha256, which botocore's S3SigV4Auth sends and its generic SigV4Auth doesn't (the signed body):

export OBJ=http://localhost:9000
python3 - <<'EOF'
import boto3, json, os
from botocore.awsrequest import AWSRequest
import botocore.auth
from urllib.request import Request, urlopen
creds = boto3.Session().get_credentials()
body = json.dumps({"pattern": "ERROR.*timeout", "max_matches": 20}).encode()
r = AWSRequest(method="POST", url=f"{os.environ['OBJ']}/logs/app.log?grep",
               data=body, headers={"Content-Type": "application/json"})
botocore.auth.S3SigV4Auth(creds, "s3", "us-east-1").add_auth(r)
with urlopen(Request(r.url, data=body, headers=dict(r.headers), method="POST")) as resp:
    for line in resp: print(line.decode().rstrip())
EOF