build · oxidb v0.25.21 0 entries on disk
The /dev/oxide

A build log on shipping OxiDB — notes, post-mortems, and the occasional flame war about JSON parsing, pressed straight onto an embedded engine running inside this process.

posts/0005.md · 2026-05-08

S3 without the IAM dance — a blob bucket inside the engine

hero image for: S3 without the IAM dance — a blob bucket inside the engine
asset · bucket: blog-images · key: a8d032e7f57a0de0f4f27f45.jpg

The blog images on this site are stored exactly the same way an S3 bucket stores objects, except the bucket is a directory inside the OxiDB data dir and the API is a Python function call instead of an AWS SDK.

Layout per object is two files in `_blobs/<bucket>/`:

<key>.data raw payload (optionally zstd-compressed)

<key>.meta JSON: content-type, etag, metadata, size

Etag is the CRC32 of the data. List/Head/Get/Put/Delete all exist on the same `OxiDb` handle as the document API. Buckets are top-level, like S3 — `create_bucket(name)`, `list_buckets()`, `delete_bucket(name)`.

The S3 wire side speaks the AWS HTTP API when `OXIDB_S3_PORT` is set. Recent additions:

* **aws-chunked decode.** AWS CLI and boto3 send streaming PUTs with `content-encoding: aws-chunked` or `x-amz-content-sha256: STREAMING-*`. Without decoding, the chunk-size headers were ending up in the stored object. The framing is now stripped back to the original payload, in both single PUT and multipart upload paths. Missing trailers and partial reads are tolerated.

* **Skip zstd for already-compressed types.** Re-compressing image/video/audio buys nothing while costing CPU on every Put and Get. The encoder now detects the content-type prefix and stores raw bytes for those. Decode stays forward-compatible with legacy zstd-stored blobs.

* **SSE.** Optional AES-256-GCM at rest, with both SSE-S3 (server-managed key) and SSE-C (per-request key) styles. `OXIDB_S3_DEFAULT_ENCRYPTION=true` encrypts every object without the client having to know.

For this blog the embedded mode is enough — no S3 port, just `db.put_object('blog-images', key, data, content_type=...)` from the Django view that handles uploads.