warehouse

warehouse is a self-contained, S3-compatible object storage server aimed at homelab and machine-learning pipelines. It is a single Go binary that stores metadata in SQLite and object content on the local filesystem, with no external services to run or configure. It speaks the AWS S3 REST API, so the standard aws CLI and any S3 SDK work against it without adapters.

Features

• S3 REST API compatibility — buckets, objects, ListBuckets, ListObjectsV2 (prefix, delimiter, max-keys, continuation-token, start-after), HEAD/GET with Range support, multipart upload (initiate / upload part / complete / abort), and idempotent deletes.
• AWS Signature V4 authentication — header (Authorization) and presigned URL variants. No anonymous/public access; every route except /healthz requires a valid signature from a known, active access key.
• Single binary, two entrypoints — warehouse runs the server; warehouse-admin manages access keys against the same metadata DB.
• CGO-free SQLite metadata via modernc.org/sqlite, so the binary is statically buildable and trivially portable.
• Flat filesystem content store — object bytes live at data/<bucket>/<key>; multipart parts stage at data/_multipart/<uploadID>/<partNumber>.
• Atomic writes — every Put and part upload writes to a temp file then renames into place, so a crash mid-write never leaves a partial object that reads would serve.
• Streaming ETags — MD5 digests are computed as bytes flow through, so large objects are not buffered. Multipart objects carry the S3 multipart ETag (MD5 of the concatenated part MD5s).
• Idempotent, auto-applying schema — the SQLite schema is embedded and applied on startup using CREATE ... IF NOT EXISTS.
• Single-writer-safe — one serialized SQLite connection; per-key writes in the content store are mutex-guarded so overlapping Puts to the same key do not interleave.

Build

Requires Go 1.26.3 or newer.

go build -o warehouse ./cmd/warehouse
go build -o warehouse-admin ./cmd/warehouse-admin

Or build everything at once:

go build ./...

Run the test suite:

go test ./...

Mage

Mage targets wrap the same commands; mage -l lists them.

mage build      # both binaries into bin/
mage test       # full suite, integration tests included
mage race       # suite under the race detector
mage fmt        # gofmt -w
mage lint       # golangci-lint
mage ci         # the gate: gofmt check, vet, lint, tests

mage ci is what the Forgejo workflow in .forgejo/workflows/ci.yml runs, so the same gate applies locally and in CI and the two cannot drift. Run it before pushing — go build and go test on their own miss the lint pass, which is what catches unchecked errors and deprecated calls. It needs golangci-lint on PATH:

go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest

Quick start

This walkthrough creates an access key, starts the server, creates a bucket, and uploads/downloads a file with the aws CLI.

1. Create an access key. The server authenticates requests with AWS SigV4, so you need an access key id and secret stored in the metadata DB. Generate one with warehouse-admin:

./warehouse-admin access-key create

This prints an AKIA... access key id, a secret access key, and an owner label. The secret is shown once; store it securely. Other admin commands:

./warehouse-admin access-key list
./warehouse-admin access-key delete <ACCESS_KEY_ID>

warehouse-admin defaults to the DB at data/warehouse.db; override with --db <path> and tag the key with --owner <name> on create.

2. Start the server.

./warehouse -addr :8080 -db data/warehouse.db -data .

Or put those values in warehouse.toml (see warehouse.toml.example) and run ./warehouse -config warehouse.toml. On startup the schema is auto-applied to the SQLite DB and the content root's data/ directory is created. Liveness is available at http://localhost:8080/healthz.

3. Use the aws CLI. Configure a profile pointing at the server with your new key:

aws --endpoint-url http://localhost:8080 \
    --region us-east-1 \
    s3 mb s3://my-bucket

aws --endpoint-url http://localhost:8080 \
    --region us-east-1 \
    s3 cp ./local-file.txt s3://my-bucket/path/to/object.txt

aws --endpoint-url http://localhost:8080 \
    --region us-east-1 \
    s3 cp s3://my-bucket/path/to/object.txt ./downloaded.txt

aws --endpoint-url http://localhost:8080 \
    --region us-east-1 \
    s3 ls s3://my-bucket/

Provide the access key id and secret through environment variables or a named profile (e.g. AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY), as you would for any S3-compatible endpoint. The region is arbitrary but must be consistent between signing and the server's SigV4 check; us-east-1 is the safe default.

Architecture overview

                 ┌─────────────────────────────────────────────┐
                 │              cmd/warehouse (server)          │
                 │  flags → open db → open store → server.New   │
                 └────────────────────┬────────────────────────┘
                                      │
            ┌─────────────────────────┴──────────────────────────┐
            │              internal/server (chi router)           │
            │  /healthz (public)                                 │
            │  /  GET  ListBuckets                                │
            │  /{bucket}      HEAD/GET/PUT/DELETE                 │
            │  /{bucket}/{key} PUT/POST/HEAD/GET/DELETE            │
            │  SigV4 auth middleware wraps every non-health route │
            └──────────┬───────────────────────────┬──────────────┘
                       │                            │
            ┌──────────▼──────────┐      ┌─────────▼────────────┐
            │  internal/auth        │      │  internal/store      │
            │  SigV4 verify         │      │  flat FS backend     │
            │  (header + presigned)  │      │  data/<bucket>/<key> │
            └──────────────────────┘      └──────────────────────┘
                       │                            │
            ┌──────────▼──────────┐      ┌─────────▼────────────┐
            │  internal/db         │      │  local filesystem     │
            │  SQLite metadata     │      │  (object content)    │
            │  (sqlx + modernc)    │      └──────────────────────┘
            └─────────────────────┘

• Metadata (internal/db) — SQLite holds buckets, objects (with versioning/deleted-at columns), in-progress multipart uploads, parts, and access keys. A single serialized connection serializes writes.
• Content (internal/store) — object bytes live on the local filesystem under <data-root>/data/<bucket>/<key>. There is no sharding or content-addressing; the path mirrors the (bucket, key) tuple directly. Multipart parts stage under data/_multipart/<uploadID>/<partNumber> and are concatenated into the final object on completion.
• HTTP (internal/server) — a chi router maps S3 operations onto the path-style URL space (/{bucket}/{key}, with {key} a wildcard that may contain slashes). All responses are S3 XML; errors use the S3 Error XML envelope with Code, Message, Resource, and RequestId.
• Auth (internal/auth) — transport-agnostic SigV4 verification against a supplied secret. The middleware does the access-key lookup and maps auth failures to S3 error codes (AccessDenied, SignatureDoesNotMatch, etc.).

API compatibility

warehouse targets the AWS S3 REST API as exercised by the aws CLI and the common S3 SDKs, using path-style addressing (http://host/<bucket>/<key>). It implements the operations most pipelines need:

• Service: ListBuckets (GET /).
• Bucket: CreateBucket (PUT /<bucket>), HeadBucket (HEAD /<bucket>), DeleteBucket (DELETE /<bucket>, refused with 409 BucketNotEmpty while the bucket still holds objects or in-progress multipart uploads), ListObjectsV2 (GET /<bucket>?list-type=2 with prefix, delimiter, max-keys, start-after, and continuation-token). Prefixes match literally and case-sensitively, and max-keys bounds the whole response — a rolled-up CommonPrefixes entry counts as a returned key, as it does in S3. The legacy v1 GET /<bucket> listing returns a minimal envelope for compatibility.
• Object: PutObject (PUT /<bucket>/<key>), GetObject (GET with single Range support), HeadObject (HEAD), DeleteObject (DELETE, idempotent).
• Multipart: CreateMultipartUpload (POST /<bucket>/<key>?uploads), UploadPart (PUT ...?partNumber=N&uploadId=ID), CompleteMultipartUpload (POST ...?uploadId=ID), AbortMultipartUpload (DELETE ...?uploadId=ID).

Object keys may contain slashes (photos/2024/img.png) and are stored in their decoded form, so a key with spaces or non-ASCII characters round-trips exactly as the client named it. Because the content store maps a key directly onto a filesystem path, keys are rejected (400) when they contain a . or .. segment, or an empty segment — a leading or trailing /, or a // run. That last rule means trailing-slash "folder marker" keys (photos/) are not supported: on a flat layout they would collide with the directory holding photos/cat.png.

Authentication is AWS Signature V4 only (header or presigned URL). There is no anonymous/public path — only /healthz is unauthenticated. Requests must be signed within 15 minutes of the server's clock, and a body signed with a concrete x-amz-content-sha256 digest is verified against that digest as it streams, so a captured signed request cannot be replayed or resent with substituted content.

Chunked uploads (Content-Encoding: aws-chunked) are decoded, which matters more than it sounds: the AWS SDKs send them by default for PutObject over HTTPS, so this is the ordinary upload path whenever warehouse sits behind a TLS-terminating reverse proxy. All three payload sentinels are handled, and each is held to what it commits to:

• STREAMING-AWS4-HMAC-SHA256-PAYLOAD (and its trailer variant) signs every chunk, chaining each signature onto the previous one and ultimately onto the request signature. The whole chain is verified, so chunks cannot be altered, reordered, or spliced between requests.
• STREAMING-UNSIGNED-PAYLOAD-TRAILER carries no chunk signatures — the client declined to sign the payload — but ends with a checksum trailer, which is verified against the decoded bytes. crc32, crc32c, crc64nvme, sha1, and sha256 are supported; a trailer naming an algorithm warehouse does not implement is accepted without that check, since it guards against corruption rather than tampering.

A STREAMING-* sentinel warehouse does not recognize is refused with 501 NotImplemented rather than stored with its framing embedded in the object.

This is a focused S3-compatible server, not a full reimplementation: features such as object versioning (beyond the latest-row model), bucket policies, server-side encryption, lifecycle rules, object lock, and replication are intentionally out of scope.

Configuration

warehouse is configured with command-line flags, optionally layered over a TOML config file:

Flag	Default	Description
-config	(none)	Path to a TOML config file. Values there override the defaults; the flags below override the file.
-addr	:8080	HTTP listen address.
-db	data/warehouse.db	SQLite database path. Use :memory: for an ephemeral in-process DB.
-data	.	Object content root. Objects are stored under <root>/data/<bucket>/<key>.

A flag only overrides the file when it is passed explicitly, so you can set common values in the file and override a single one on a given run. Precedence is: CLI flags > config file > built-in defaults. See warehouse.toml.example in the repo root for a documented template:

cp warehouse.toml.example warehouse.toml
./warehouse -config warehouse.toml

warehouse-admin shares the -db flag (same default) and adds -owner for access-key create. Run with -h for the full usage.

The SQLite connection is opened with foreign_keys(1), a busy_timeout of 5000ms, and WAL journal mode.

Timeouts

warehouse deliberately sets no ReadTimeout or WriteTimeout on its HTTP server, and no blanket request timeout. Those are absolute deadlines covering the request and response bodies, so on an object store they do not bound misbehavior — they bound object size. A 30s write timeout makes any object that cannot be sent in 30 seconds permanently unfetchable, however healthy the connection.

What is bounded instead is a lack of progress. A single read of a request body or write of a response body may block for up to 60s; a transfer as a whole may take as long as its size warrants. Requests headers get 10s, idle keep-alive connections 120s, and metadata queries — whose cost does not scale with object size — 10s.

The one operational consequence: because a blocked socket writer is not rewoken until roughly half the send buffer drains, a client that sustains less than roughly 34 KB/s may be dropped mid-download. A consumer that slow would need over eight hours per gigabyte.