feat: Presigned object URLs (s3 driver)

POST /buckets/:id/object-presigns mints SigV4 query-signed GET/PUT
URLs so clients move bytes directly against S3 — large transfers skip
the backend entirely.

- storage.Presigner optional interface; S3Store implements, local
  driver returns a clear error (no external surface to sign).
- TTL 60s–1h (default 15m); download requires read role + existing
  object row (no key probing); upload requires write role.
- S3_PUBLIC_ENDPOINT rewrites the client-facing host for internal
  S3_ENDPOINTs — signatures bind the host, URLs are valid only where
  minted.
- CLI: primora objects:presign <bucket> <key> [--upload] [--ttl].

Verified live against cloudserver: GET fetched the object with no
auth headers; PUT pushed 1 MB direct to S3; unsigned list → 403;
untracked-key download presign → 404.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Tomas DvorakandDevin committed 2026-09-24 20:35:21 +02:00
1 parent a84049b8fc
commit fd36389328
22 files changed
+446 -7

No files matched your search

+4
View File
@@ -60,6 +60,10 @@ BACKEND_STORAGE_DRIVER=local
#S3_SECRET_ACCESS_KEY= #S3_SECRET_ACCESS_KEY=
#S3_PREFIX= #S3_PREFIX=
#S3_PATH_STYLE=true #S3_PATH_STYLE=true
# Client-facing S3 endpoint for presigned URLs — set when S3_ENDPOINT is an
# internal address (e.g. http://minio:9000 inside compose) that browsers
# can't reach. Presigned URLs are valid only at the host they were minted for.
#S3_PUBLIC_ENDPOINT=https://objects.example.com
# Mail defaults — each is overridable in-app (Settings → Instance). In-app # Mail defaults — each is overridable in-app (Settings → Instance). In-app
# values win over these env vars; deleting the in-app value reverts here. # values win over these env vars; deleting the in-app value reverts here.
+1
View File
@@ -25,6 +25,7 @@ All notable changes to Primora. Format follows [Keep a Changelog](https://keepac
- **Email templates + send log** (`core.email_log`, migration 00014) — the mailer renders registered templates (currently `invitation`) and every send — successful or failed — is recorded with template, recipient, subject, status, and error text. `GET /projects/:id/emails` lists recent sends; the Members page shows an "Email log" card under pending invitations. - **Email templates + send log** (`core.email_log`, migration 00014) — the mailer renders registered templates (currently `invitation`) and every send — successful or failed — is recorded with template, recipient, subject, status, and error text. `GET /projects/:id/emails` lists recent sends; the Members page shows an "Email log" card under pending invitations.
- **S3-compatible object storage backend** — `BACKEND_STORAGE_DRIVER=s3` swaps the local disk store for any S3-compatible API (AWS, MinIO, Garage, R2) via a stdlib-only SigV4 implementation — no AWS SDK dependency. `S3_ENDPOINT`/`S3_REGION`/`S3_BUCKET`/credentials configure it; `S3_PATH_STYLE` (default on) covers MinIO/Garage, off for AWS virtual-hosted. The configured bucket is created on startup if missing; objects stream with computed SHA-256, `storage_path` records `s3://bucket/key`. Verified live end-to-end (bucket → put → get → copy → list → delete). - **S3-compatible object storage backend** — `BACKEND_STORAGE_DRIVER=s3` swaps the local disk store for any S3-compatible API (AWS, MinIO, Garage, R2) via a stdlib-only SigV4 implementation — no AWS SDK dependency. `S3_ENDPOINT`/`S3_REGION`/`S3_BUCKET`/credentials configure it; `S3_PATH_STYLE` (default on) covers MinIO/Garage, off for AWS virtual-hosted. The configured bucket is created on startup if missing; objects stream with computed SHA-256, `storage_path` records `s3://bucket/key`. Verified live end-to-end (bucket → put → get → copy → list → delete).
- **Functions** (`core.functions` + `core.function_runs`, migration 00015) — project-scoped JS/TS executed by a runtime binary on the backend host (`bun` or `deno`, per function). The JSON payload arrives on stdin and as `PRIMORA_PAYLOAD`; stdout/stderr are captured per run (capped at 64 KiB) with status, exit code and duration — `FUNCTIONS_RUNTIME`/`FUNCTIONS_TIMEOUT_SECONDS`/`FUNCTIONS_MAX_OUTPUT_BYTES` configure the runner. API: CRUD under `/projects/:id/functions`, `POST …/invoke` (502 + recorded run on failure), `GET …/runs`. Dashboard gains a Functions page: code editor, JSON payload invoke, live result, run history. Code executes with backend privileges — the documented self-hosted trust model. - **Functions** (`core.functions` + `core.function_runs`, migration 00015) — project-scoped JS/TS executed by a runtime binary on the backend host (`bun` or `deno`, per function). The JSON payload arrives on stdin and as `PRIMORA_PAYLOAD`; stdout/stderr are captured per run (capped at 64 KiB) with status, exit code and duration — `FUNCTIONS_RUNTIME`/`FUNCTIONS_TIMEOUT_SECONDS`/`FUNCTIONS_MAX_OUTPUT_BYTES` configure the runner. API: CRUD under `/projects/:id/functions`, `POST …/invoke` (502 + recorded run on failure), `GET …/runs`. Dashboard gains a Functions page: code editor, JSON payload invoke, live result, run history. Code executes with backend privileges — the documented self-hosted trust model.
- **Presigned object URLs** — `POST /buckets/:bucketID/object-presigns` mints SigV4 query-signed `GET`/`PUT` URLs (60 s–1 h TTL) so clients transfer large objects directly against S3 without proxying through the backend. Read role for downloads (object must exist — no key probing), write role for uploads. `S3_PUBLIC_ENDPOINT` rewrites the client-facing host for internal `S3_ENDPOINT`s; signatures bind the host so URLs are valid only where they were minted. The local driver returns a clear error — a presigned URL needs a real object API.
- **Function triggers** (migration 00016) — three new ways to run a function, all through the same executor and run log. Scheduled jobs accept `function_id` instead of `url` (run status mirrors the exit code); inbound hooks gain `mode: "function"` — the received body becomes the function's stdin payload, acked immediately with the run recorded async; and `event_pattern` on a function (`document.*`, `*`, or an exact type) fires it detached whenever a matching domain event publishes — `{event, project_id, occurred_at, data}` as payload. Runs record trigger source: `manual`/`schedule`/`hook`/`event`. Dashboard: function event-trigger field, job target picker (URL or function), and inbound-hook function mode. - **Function triggers** (migration 00016) — three new ways to run a function, all through the same executor and run log. Scheduled jobs accept `function_id` instead of `url` (run status mirrors the exit code); inbound hooks gain `mode: "function"` — the received body becomes the function's stdin payload, acked immediately with the run recorded async; and `event_pattern` on a function (`document.*`, `*`, or an exact type) fires it detached whenever a matching domain event publishes — `{event, project_id, occurred_at, data}` as payload. Runs record trigger source: `manual`/`schedule`/`hook`/`event`. Dashboard: function event-trigger field, job target picker (URL or function), and inbound-hook function mode.
## [0.7.1] - 2026-09-23 ## [0.7.1] - 2026-09-23
+1
View File
@@ -46,6 +46,7 @@ docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d
| `S3_ENDPOINT` / `S3_REGION` / `S3_BUCKET` | S3 API endpoint (empty endpoint → AWS `s3.<region>.amazonaws.com`), region, bucket | with `s3` | | `S3_ENDPOINT` / `S3_REGION` / `S3_BUCKET` | S3 API endpoint (empty endpoint → AWS `s3.<region>.amazonaws.com`), region, bucket | with `s3` |
| `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | S3 credentials | with `s3` | | `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | S3 credentials | with `s3` |
| `S3_PREFIX` / `S3_PATH_STYLE` | optional key prefix; path-style URLs (default `true`, needed by MinIO/Garage — set `false` for AWS virtual-hosted) | no | | `S3_PREFIX` / `S3_PATH_STYLE` | optional key prefix; path-style URLs (default `true`, needed by MinIO/Garage — set `false` for AWS virtual-hosted) | no |
| `S3_PUBLIC_ENDPOINT` | client-facing S3 endpoint used for presigned URLs when `S3_ENDPOINT` is internal-only (e.g. `http://minio:9000`); signatures bind the public host | no |
| `FUNCTIONS_RUNTIME` / `FUNCTIONS_TIMEOUT_SECONDS` / `FUNCTIONS_MAX_OUTPUT_BYTES` | Functions: runtime binary (`auto` resolves `bun`/`deno` from the backend's PATH — the backend image must ship one, or set this to its path), per-run timeout (default 30 s), captured-output cap (default 64 KiB) | no | | `FUNCTIONS_RUNTIME` / `FUNCTIONS_TIMEOUT_SECONDS` / `FUNCTIONS_MAX_OUTPUT_BYTES` | Functions: runtime binary (`auto` resolves `bun`/`deno` from the backend's PATH — the backend image must ship one, or set this to its path), per-run timeout (default 30 s), captured-output cap (default 64 KiB) | no |
| `AUTH_ADMIN_EMAILS` | comma-separated emails promoted to auth admin on boot | no | | `AUTH_ADMIN_EMAILS` | comma-separated emails promoted to auth admin on boot | no |
| `USER_RATE_LIMIT_PER_MINUTE` / `API_KEY_RATE_LIMIT_PER_MINUTE` | per-identity API rate limits (defaults 240 / 600) | no | | `USER_RATE_LIMIT_PER_MINUTE` / `API_KEY_RATE_LIMIT_PER_MINUTE` | per-identity API rate limits (defaults 240 / 600) | no |
+1
View File
@@ -67,6 +67,7 @@ The important ones:
| `USER_RATE_LIMIT_PER_MINUTE` / `API_KEY_RATE_LIMIT_PER_MINUTE` | Throttling defaults — overridable in-app | | `USER_RATE_LIMIT_PER_MINUTE` / `API_KEY_RATE_LIMIT_PER_MINUTE` | Throttling defaults — overridable in-app |
| `BACKEND_STORAGE_DRIVER` | `local` (default, `BACKEND_STORAGE_ROOT`) or `s3` — S3-compatible object storage | | `BACKEND_STORAGE_DRIVER` | `local` (default, `BACKEND_STORAGE_ROOT`) or `s3` — S3-compatible object storage |
| `S3_ENDPOINT` / `S3_BUCKET` / `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | S3 backend config (with `S3_REGION`, `S3_PREFIX`, `S3_PATH_STYLE`) | | `S3_ENDPOINT` / `S3_BUCKET` / `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | S3 backend config (with `S3_REGION`, `S3_PREFIX`, `S3_PATH_STYLE`) |
| `S3_PUBLIC_ENDPOINT` | Client-facing S3 host for presigned URLs when `S3_ENDPOINT` is internal-only |
| `FUNCTIONS_RUNTIME` / `FUNCTIONS_TIMEOUT_SECONDS` | JS runtime for Functions — `auto` (default, resolves `bun`/`deno` from PATH) or a binary path; 30 s default timeout | | `FUNCTIONS_RUNTIME` / `FUNCTIONS_TIMEOUT_SECONDS` | JS runtime for Functions — `auto` (default, resolves `bun`/`deno` from PATH) or a binary path; 30 s default timeout |
| `VITE_DEMO_MODE` | Enable the client-side demo workspace | | `VITE_DEMO_MODE` | Enable the client-side demo workspace |
+1 -1
View File
@@ -48,7 +48,7 @@ Supabase, with no hosted tier - your data stays on your hardware.
- **Organizations & projects** - workspaces with scoped org and project roles. - **Organizations & projects** - workspaces with scoped org and project roles.
- **Auth** - email/password plus optional GitHub, Google, Discord, and Microsoft OAuth. JWTs verified by the API against JWKS. - **Auth** - email/password plus optional GitHub, Google, Discord, and Microsoft OAuth. JWTs verified by the API against JWKS.
- **Storage** - S3-style buckets on local disk or any S3-compatible backend (AWS, MinIO, Garage, R2), public or private. - **Storage** - S3-style buckets on local disk or any S3-compatible backend (AWS, MinIO, Garage, R2), public or private. With `s3` the API mints presigned upload/download URLs so large objects skip the backend.
- **Collections** - schema-flexible JSON documents in Postgres JSONB, with filter/order queries. - **Collections** - schema-flexible JSON documents in Postgres JSONB, with filter/order queries.
- **Functions** - project-scoped JS/TS executed by bun/deno on the host, with per-run logs. Trigger them four ways: manual invoke, cron schedules (`function_id` on a job), inbound hooks (`mode: "function"`), or domain events (`event_pattern` like `document.*`). - **Functions** - project-scoped JS/TS executed by bun/deno on the host, with per-run logs. Trigger them four ways: manual invoke, cron schedules (`function_id` on a job), inbound hooks (`mode: "function"`), or domain events (`event_pattern` like `document.*`).
- **Vault** - AES-256-GCM project secrets plus an Argon2id/XChaCha20 local vault in the CLI and desktop app. - **Vault** - AES-256-GCM project secrets plus an Argon2id/XChaCha20 local vault in the CLI and desktop app.
+6 -4
View File
@@ -365,10 +365,12 @@ Ordered loosely by leverage. None committed; each gets scoped when picked.
Runs record trigger source (`manual`/`schedule`/`hook`/`event`). Runs record trigger source (`manual`/`schedule`/`hook`/`event`).
Remaining candidate: isolated runtimes (sidecar/Firecracker) for Remaining candidate: isolated runtimes (sidecar/Firecracker) for
untrusted code — functions still execute with backend host privileges. untrusted code — functions still execute with backend host privileges.
- **Phase 10 — Storage backends** — ✅ S3-compatible object storage shipped: - **Phase 10 — Storage backends** — ✅ shipped: `BACKEND_STORAGE_DRIVER=s3`
`BACKEND_STORAGE_DRIVER=s3` + `S3_*` env vars, stdlib SigV4, verified + `S3_*` env vars, stdlib SigV4, verified against a real S3 server.
against a real S3 server. Presigned-URL uploads for large files remain a Presigned upload/download URLs shipped: `POST
candidate. /buckets/:id/object-presigns` mints SigV4 query-signed GET/PUT URLs
(`S3_PUBLIC_ENDPOINT` rewrites the client-facing host); local driver
returns a clear error — a presigned URL needs a real object API behind it.
- **HA mode** — ✅ shipped: PostgreSQL advisory-lock leadership for the - **HA mode** — ✅ shipped: PostgreSQL advisory-lock leadership for the
scheduler and alert evaluator; failover verified with two replicas. scheduler and alert evaluator; failover verified with two replicas.
- **Email surface** — ✅ shipped: templated sends + `core.email_log` + - **Email surface** — ✅ shipped: templated sends + `core.email_log` +
+1
View File
@@ -89,6 +89,7 @@ func Bootstrap(ctx context.Context) (*App, error) {
Bucket: cfg.S3Bucket, Bucket: cfg.S3Bucket,
AccessKeyID: cfg.S3AccessKeyID, AccessKeyID: cfg.S3AccessKeyID,
SecretAccessKey: cfg.S3SecretAccessKey, SecretAccessKey: cfg.S3SecretAccessKey,
PublicEndpoint: cfg.S3PublicEndpoint,
Prefix: cfg.S3Prefix, Prefix: cfg.S3Prefix,
PathStyle: cfg.S3PathStyle, PathStyle: cfg.S3PathStyle,
}) })
+2
View File
@@ -44,6 +44,7 @@ type Config struct {
S3Bucket string S3Bucket string
S3AccessKeyID string S3AccessKeyID string
S3SecretAccessKey string S3SecretAccessKey string
S3PublicEndpoint string // presigned-URL host when it differs from Endpoint
S3Prefix string S3Prefix string
S3PathStyle bool S3PathStyle bool
@@ -86,6 +87,7 @@ func Load() (Config, error) {
S3Bucket: os.Getenv("S3_BUCKET"), S3Bucket: os.Getenv("S3_BUCKET"),
S3AccessKeyID: os.Getenv("S3_ACCESS_KEY_ID"), S3AccessKeyID: os.Getenv("S3_ACCESS_KEY_ID"),
S3SecretAccessKey: os.Getenv("S3_SECRET_ACCESS_KEY"), S3SecretAccessKey: os.Getenv("S3_SECRET_ACCESS_KEY"),
S3PublicEndpoint: os.Getenv("S3_PUBLIC_ENDPOINT"),
S3Prefix: os.Getenv("S3_PREFIX"), S3Prefix: os.Getenv("S3_PREFIX"),
FunctionsRuntime: getenv("FUNCTIONS_RUNTIME", "auto"), FunctionsRuntime: getenv("FUNCTIONS_RUNTIME", "auto"),
FunctionsMaxBytes: 65536, FunctionsMaxBytes: 65536,
+34
View File
@@ -142,6 +142,7 @@ func (h *HTTPHandler) Register(router *gin.Engine) {
api.GET("/buckets/:bucketID/objects", h.listObjects) api.GET("/buckets/:bucketID/objects", h.listObjects)
api.POST("/buckets/:bucketID/objects", h.uploadObject) api.POST("/buckets/:bucketID/objects", h.uploadObject)
api.POST("/buckets/:bucketID/object-copies", h.copyObject) api.POST("/buckets/:bucketID/object-copies", h.copyObject)
api.POST("/buckets/:bucketID/object-presigns", h.presignObject)
api.GET("/buckets/:bucketID/objects/*objectKey", h.downloadObject) api.GET("/buckets/:bucketID/objects/*objectKey", h.downloadObject)
api.PATCH("/buckets/:bucketID/objects/*objectKey", h.updateObject) api.PATCH("/buckets/:bucketID/objects/*objectKey", h.updateObject)
api.DELETE("/buckets/:bucketID/objects/*objectKey", h.deleteObject) api.DELETE("/buckets/:bucketID/objects/*objectKey", h.deleteObject)
@@ -891,6 +892,39 @@ func (h *HTTPHandler) updateObject(c *gin.Context) {
c.Status(http.StatusNoContent) c.Status(http.StatusNoContent)
} }
type presignObjectRequest struct {
Key string `json:"key" validate:"required"`
Op string `json:"op" validate:"omitempty,oneof=download upload"`
TTLSeconds int `json:"ttl_seconds" validate:"omitempty,min=60,max=3600"`
}
// presignObject returns a short-lived URL clients call against storage
// directly — keeps large transfers off the backend. S3 driver only.
func (h *HTTPHandler) presignObject(c *gin.Context) {
actor, ok := middleware.RequireActor(c)
if !ok {
return
}
bucketID, ok := parseUUIDParam(c, "bucketID")
if !ok {
return
}
var body presignObjectRequest
if !h.bindAndValidate(c, &body) {
return
}
out, err := h.Platform.PresignObject(c.Request.Context(), actor, bucketID, strings.TrimSpace(body.Key), body.Op, body.TTLSeconds)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
apperrors.Abort(c, http.StatusNotFound, "not_found", "bucket or object not found")
return
}
h.handleError(c, err)
return
}
c.JSON(http.StatusOK, out)
}
func (h *HTTPHandler) deleteObject(c *gin.Context) { func (h *HTTPHandler) deleteObject(c *gin.Context) {
actor, ok := middleware.RequireActor(c) actor, ok := middleware.RequireActor(c)
if !ok { if !ok {
@@ -10,6 +10,7 @@ import (
"fmt" "fmt"
"io" "io"
"log/slog" "log/slog"
"net/http"
"net/url" "net/url"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -1418,6 +1419,48 @@ func (s *PlatformService) GetObject(ctx context.Context, actor *models.Actor, bu
return object, file, nil return object, file, nil
} }
// PresignObject mints a short-lived direct-to-storage URL for a single
// object. Requires the s3 storage driver — the local store has no external
// URL surface to sign against.
func (s *PlatformService) PresignObject(ctx context.Context, actor *models.Actor, bucketID uuid.UUID, objectKey, op string, ttlSeconds int) (*storage.PresignedURL, error) {
bucket, err := s.repo.Queries().GetBucketByID(ctx, bucketID)
if err != nil {
return nil, err
}
method := ""
switch op {
case "download", "":
method = http.MethodGet
if err := s.requireBucketRead(ctx, actor, bucket); err != nil {
return nil, err
}
if !objectExists(ctx, s, bucketID, objectKey) {
return nil, pgx.ErrNoRows
}
case "upload":
method = http.MethodPut
if err := s.requireBucketWrite(ctx, actor, bucket); err != nil {
return nil, err
}
default:
return nil, fmt.Errorf("invalid op %q — download or upload", op)
}
p, ok := s.store.(storage.Presigner)
if !ok {
return nil, fmt.Errorf("presigned URLs require BACKEND_STORAGE_DRIVER=s3")
}
ttl := time.Duration(ttlSeconds) * time.Second
return p.Presign(ctx, bucketID.String(), objectKey, method, ttl)
}
func objectExists(ctx context.Context, s *PlatformService, bucketID uuid.UUID, key string) bool {
_, err := s.repo.Queries().GetBucketObjectByKey(ctx, db.GetBucketObjectByKeyParams{
BucketID: bucketID,
ObjectKey: key,
})
return err == nil
}
func (s *PlatformService) UpdateObject(ctx context.Context, actor *models.Actor, bucketID uuid.UUID, objectKey string, input UpdateObjectInput, requestID string) (db.CoreBucketObject, error) { func (s *PlatformService) UpdateObject(ctx context.Context, actor *models.Actor, bucketID uuid.UUID, objectKey string, input UpdateObjectInput, requestID string) (db.CoreBucketObject, error) {
sourceBucket, err := s.repo.Queries().GetBucketByID(ctx, bucketID) sourceBucket, err := s.repo.Queries().GetBucketByID(ctx, bucketID)
if err != nil { if err != nil {
+87 -1
View File
@@ -28,7 +28,11 @@ type S3Config struct {
SecretAccessKey string SecretAccessKey string
Prefix string // optional key prefix, e.g. "primora/" Prefix string // optional key prefix, e.g. "primora/"
PathStyle bool // bucket/key path style (MinIO, Garage) vs virtual-hosted PathStyle bool // bucket/key path style (MinIO, Garage) vs virtual-hosted
HTTPClient *http.Client // PublicEndpoint rewrites presigned-URL hosts for clients that can't
// reach Endpoint (e.g. minio:9000 inside compose). Signature binds the
// public host — the URL is only valid there.
PublicEndpoint string
HTTPClient *http.Client
} }
// S3Store implements Store against an S3-compatible API using stdlib-only // S3Store implements Store against an S3-compatible API using stdlib-only
@@ -271,6 +275,88 @@ func (s *S3Store) DeleteBucket(bucketID string) error {
} }
} }
// Presign mints a SigV4 query-signed URL for a single GET/PUT on the
// object — clients talk to S3 directly instead of proxying bytes through
// the backend. TTL is capped at one hour.
func (s *S3Store) Presign(ctx context.Context, bucketID, objectKey, method string, ttl time.Duration) (*PresignedURL, error) {
if method != http.MethodGet && method != http.MethodPut {
return nil, fmt.Errorf("s3 presign: unsupported method %q", method)
}
key, err := s.objectKey(bucketID, objectKey)
if err != nil {
return nil, err
}
if ttl <= 0 {
ttl = 15 * time.Minute
}
if ttl > time.Hour {
ttl = time.Hour
}
now := s.now().UTC()
amzDate := now.Format("20060102T150405Z")
dateStamp := now.Format("20060102")
scope := fmt.Sprintf("%s/%s/s3/aws4_request", dateStamp, s.cfg.Region)
// Build the unsigned URL — same path/vhost logic as signedRequest, but
// on the client-facing endpoint when one is configured.
var base *url.URL
endpoint := strings.TrimSuffix(s.cfg.PublicEndpoint, "/")
if endpoint == "" {
endpoint = strings.TrimSuffix(s.cfg.Endpoint, "/")
}
if s.cfg.PathStyle {
u, err := url.Parse(endpoint + "/" + s.cfg.Bucket + "/" + escapeKey(key))
if err != nil {
return nil, fmt.Errorf("s3 presign url: %w", err)
}
base = u
} else {
u, err := url.Parse(endpoint)
if err != nil {
return nil, fmt.Errorf("s3 presign url: %w", err)
}
u.Host = s.cfg.Bucket + "." + u.Host
u.Path = "/" + escapeKey(key)
base = u
}
q := base.Query()
q.Set("X-Amz-Algorithm", "AWS4-HMAC-SHA256")
q.Set("X-Amz-Credential", s.cfg.AccessKeyID+"/"+scope)
q.Set("X-Amz-Date", amzDate)
q.Set("X-Amz-Expires", fmt.Sprintf("%d", int64(ttl.Seconds())))
q.Set("X-Amz-SignedHeaders", "host")
canonicalURI := base.EscapedPath()
if canonicalURI == "" {
canonicalURI = "/"
}
canonicalRequest := strings.Join([]string{
method,
canonicalURI,
canonicalQuery(q),
"host:" + base.Host + "\n",
"host",
"UNSIGNED-PAYLOAD",
}, "\n")
stringToSign := strings.Join([]string{
"AWS4-HMAC-SHA256",
amzDate,
scope,
hexSHA256(canonicalRequest),
}, "\n")
kDate := hmacSHA256([]byte("AWS4"+s.cfg.SecretAccessKey), dateStamp)
kRegion := hmacSHA256(kDate, s.cfg.Region)
kService := hmacSHA256(kRegion, "s3")
kSigning := hmacSHA256(kService, "aws4_request")
q.Set("X-Amz-Signature", hex.EncodeToString(hmacSHA256(kSigning, stringToSign)))
base.RawQuery = q.Encode()
return &PresignedURL{URL: base.String(), Method: method, ExpiresAt: now.Add(ttl)}, nil
}
type listResult struct { type listResult struct {
Contents []struct{ Key string `xml:"Key"` } `xml:"Contents"` Contents []struct{ Key string `xml:"Key"` } `xml:"Contents"`
NextContinuationToken string `xml:"NextContinuationToken"` NextContinuationToken string `xml:"NextContinuationToken"`
+35
View File
@@ -6,6 +6,7 @@ import (
"io" "io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -214,3 +215,37 @@ func TestCanonicalQueryRFC3986(t *testing.T) {
t.Fatalf("canonicalQuery = %q, want %q", got, want) t.Fatalf("canonicalQuery = %q, want %q", got, want)
} }
} }
func TestS3StorePresign(t *testing.T) {
store, _ := newTestS3(t)
ctx := context.Background()
dl, err := store.Presign(ctx, "bucket-a", "docs/x.txt", http.MethodGet, 10*time.Minute)
if err != nil {
t.Fatalf("Presign: %v", err)
}
if dl.Method != http.MethodGet || dl.ExpiresAt.IsZero() {
t.Fatalf("bad presign: %+v", dl)
}
u, err := url.Parse(dl.URL)
if err != nil {
t.Fatalf("url parse: %v", err)
}
q := u.Query()
for _, k := range []string{"X-Amz-Algorithm", "X-Amz-Credential", "X-Amz-Date", "X-Amz-Expires", "X-Amz-SignedHeaders", "X-Amz-Signature"} {
if q.Get(k) == "" {
t.Fatalf("missing %s in %s", k, dl.URL)
}
}
if q.Get("X-Amz-Expires") != "600" || q.Get("X-Amz-SignedHeaders") != "host" {
t.Fatalf("bad query: %s", dl.URL)
}
if _, err := store.Presign(ctx, "b", "k", http.MethodDelete, 0); err == nil {
t.Fatal("expected method rejection")
}
// TTL cap.
capped, _ := store.Presign(ctx, "b", "k.txt", http.MethodPut, 48*time.Hour)
if cu, _ := url.Parse(capped.URL); cu.Query().Get("X-Amz-Expires") != "3600" {
t.Fatalf("ttl not capped: %s", capped.URL)
}
}
+15
View File
@@ -3,6 +3,7 @@ package storage
import ( import (
"context" "context"
"io" "io"
"time"
) )
// Store is the object-storage contract the platform service depends on. // Store is the object-storage contract the platform service depends on.
@@ -14,3 +15,17 @@ type Store interface {
MoveBetweenBuckets(sourceBucketID, destinationBucketID, fromKey, toKey string) (string, error) MoveBetweenBuckets(sourceBucketID, destinationBucketID, fromKey, toKey string) (string, error)
DeleteBucket(bucketID string) error DeleteBucket(bucketID string) error
} }
// PresignedURL is a short-lived direct-to-storage URL for one operation.
type PresignedURL struct {
URL string `json:"url"`
Method string `json:"method"`
ExpiresAt time.Time `json:"expires_at"`
}
// Presigner is an optional capability: stores backed by a real object API
// (S3) can mint URLs clients call directly. The local driver can't — its
// "URL" would have to be served by this backend anyway.
type Presigner interface {
Presign(ctx context.Context, bucketID, objectKey, method string, ttl time.Duration) (*PresignedURL, error)
}
+61
View File
@@ -677,6 +677,37 @@ paths:
application/json: application/json:
schema: schema:
$ref: "#/components/schemas/BucketObject" $ref: "#/components/schemas/BucketObject"
/buckets/{bucketID}/object-presigns:
post:
tags: [Storage]
operationId: presignBucketObject
description: >-
Mint a short-lived URL the client calls against object storage
directly — keeps large transfers off the backend. Requires
BACKEND_STORAGE_DRIVER=s3; the URL's host comes from
S3_PUBLIC_ENDPOINT (or S3_ENDPOINT when unset). Signatures bind the
host — a presigned URL is valid only at the endpoint it was minted
for.
security:
- bearerAuth: []
- apiKeyAuth: []
parameters:
- $ref: "#/components/parameters/BucketID"
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/PresignObjectRequest"
responses:
"200":
description: Presigned URL
content:
application/json:
schema:
$ref: "#/components/schemas/PresignedURL"
"404":
description: Bucket or object not found (download)
/buckets/{bucketID}: /buckets/{bucketID}:
patch: patch:
tags: [Storage] tags: [Storage]
@@ -3130,6 +3161,36 @@ components:
format: uuid format: uuid
nullable: true nullable: true
required: [objectKey, newObjectKey] required: [objectKey, newObjectKey]
PresignObjectRequest:
type: object
properties:
key:
type: string
description: Object key inside the bucket.
op:
type: string
enum: [download, upload]
default: download
description: download → GET URL (object must exist); upload → PUT URL (creates/overwrites).
ttl_seconds:
type: integer
minimum: 60
maximum: 3600
default: 900
required: [key]
PresignedURL:
type: object
properties:
url:
type: string
description: Fully-signed URL — call it with the returned method, no auth header needed.
method:
type: string
enum: [GET, PUT]
expires_at:
type: string
format: date-time
required: [url, method, expires_at]
PaginationMetadata: PaginationMetadata:
type: object type: object
properties: properties:
+7
View File
@@ -12,6 +12,7 @@ import {
cmdObjectsDownload, cmdObjectsDownload,
cmdObjectsList, cmdObjectsList,
cmdObjectsRemove, cmdObjectsRemove,
cmdObjectsPresign,
cmdObjectsUpload, cmdObjectsUpload,
} from "./commands/objects.js"; } from "./commands/objects.js";
import { cmdOrgsCreate, cmdOrgsList } from "./commands/orgs.js"; import { cmdOrgsCreate, cmdOrgsList } from "./commands/orgs.js";
@@ -114,6 +115,12 @@ cli
.command("objects:rm <bucket> <key>", "Delete an object") .command("objects:rm <bucket> <key>", "Delete an object")
.option("--project <id>", "Project override") .option("--project <id>", "Project override")
.action(cmdObjectsRemove); .action(cmdObjectsRemove);
cli
.command("objects:presign <bucket> <key>", "Mint a presigned URL (s3 driver)")
.option("--project <id>", "Project override")
.option("--upload", "PUT URL for direct upload instead of GET download")
.option("--ttl <seconds>", "Lifetime 60-3600 (default 900)")
.action(cmdObjectsPresign);
cli cli
.command("keys:list", "List API keys for the current project") .command("keys:list", "List API keys for the current project")
+29 -1
View File
@@ -2,7 +2,7 @@ import { statSync } from "node:fs";
import { readFile, writeFile } from "node:fs/promises"; import { readFile, writeFile } from "node:fs/promises";
import { basename } from "node:path"; import { basename } from "node:path";
import { StorageService } from "@primora/api-client"; import { PresignObjectRequest, StorageService } from "@primora/api-client";
import { loadConfig } from "../config.js"; import { loadConfig } from "../config.js";
import { import {
@@ -133,3 +133,31 @@ export async function cmdObjectsRemove(
await StorageService.deleteBucketObject({ bucketId, objectKey }); await StorageService.deleteBucketObject({ bucketId, objectKey });
success(`Deleted ${objectKey}`); success(`Deleted ${objectKey}`);
} }
export async function cmdObjectsPresign(
bucketRef: string,
objectKey: string,
options: { project?: string; upload?: boolean; ttl?: string; json?: boolean },
): Promise<void> {
const cfg = loadConfig();
requireAuth(cfg);
configureClient(cfg);
const bucketId = await bucketIdFor(cfg, bucketRef, options.project);
const presigned = await StorageService.presignBucketObject({
bucketId,
requestBody: {
key: objectKey,
op: options.upload
? PresignObjectRequest.op.UPLOAD
: PresignObjectRequest.op.DOWNLOAD,
ttl_seconds: options.ttl ? Number(options.ttl) : undefined,
},
});
if (isJson(options)) {
printJson(presigned);
return;
}
console.log(presigned.url);
console.error(`# ${presigned.method} — expires ${presigned.expires_at}`);
}
@@ -108,6 +108,8 @@ export type { OrganizationSummary } from './models/OrganizationSummary';
export type { PaginationLimit } from './models/PaginationLimit'; export type { PaginationLimit } from './models/PaginationLimit';
export type { PaginationMetadata } from './models/PaginationMetadata'; export type { PaginationMetadata } from './models/PaginationMetadata';
export type { PaginationOffset } from './models/PaginationOffset'; export type { PaginationOffset } from './models/PaginationOffset';
export { PresignedURL } from './models/PresignedURL';
export { PresignObjectRequest } from './models/PresignObjectRequest';
export type { Project } from './models/Project'; export type { Project } from './models/Project';
export type { ProjectID } from './models/ProjectID'; export type { ProjectID } from './models/ProjectID';
export { ProjectMember } from './models/ProjectMember'; export { ProjectMember } from './models/ProjectMember';
@@ -258,6 +260,8 @@ export { $OrganizationSummary } from './schemas/$OrganizationSummary';
export { $PaginationLimit } from './schemas/$PaginationLimit'; export { $PaginationLimit } from './schemas/$PaginationLimit';
export { $PaginationMetadata } from './schemas/$PaginationMetadata'; export { $PaginationMetadata } from './schemas/$PaginationMetadata';
export { $PaginationOffset } from './schemas/$PaginationOffset'; export { $PaginationOffset } from './schemas/$PaginationOffset';
export { $PresignedURL } from './schemas/$PresignedURL';
export { $PresignObjectRequest } from './schemas/$PresignObjectRequest';
export { $Project } from './schemas/$Project'; export { $Project } from './schemas/$Project';
export { $ProjectID } from './schemas/$ProjectID'; export { $ProjectID } from './schemas/$ProjectID';
export { $ProjectMember } from './schemas/$ProjectMember'; export { $ProjectMember } from './schemas/$ProjectMember';
@@ -0,0 +1,25 @@
/* generated using openapi-typescript-codegen -- do not edit */
/* istanbul ignore file */
/* tslint:disable */
/* eslint-disable */
export type PresignObjectRequest = {
/**
* Object key inside the bucket.
*/
key: string;
/**
* download → GET URL (object must exist); upload → PUT URL (creates/overwrites).
*/
op?: PresignObjectRequest.op;
ttl_seconds?: number;
};
export namespace PresignObjectRequest {
/**
* download → GET URL (object must exist); upload → PUT URL (creates/overwrites).
*/
export enum op {
DOWNLOAD = 'download',
UPLOAD = 'upload',
}
}
@@ -0,0 +1,19 @@
/* generated using openapi-typescript-codegen -- do not edit */
/* istanbul ignore file */
/* tslint:disable */
/* eslint-disable */
export type PresignedURL = {
/**
* Fully-signed URL — call it with the returned method, no auth header needed.
*/
url: string;
method: PresignedURL.method;
expires_at: string;
};
export namespace PresignedURL {
export enum method {
GET = 'GET',
PUT = 'PUT',
}
}
@@ -0,0 +1,21 @@
/* generated using openapi-typescript-codegen -- do not edit */
/* istanbul ignore file */
/* tslint:disable */
/* eslint-disable */
export const $PresignObjectRequest = {
properties: {
key: {
type: 'string',
description: `Object key inside the bucket.`,
isRequired: true,
},
op: {
type: 'Enum',
},
ttl_seconds: {
type: 'number',
maximum: 3600,
minimum: 60,
},
},
} as const;
@@ -0,0 +1,22 @@
/* generated using openapi-typescript-codegen -- do not edit */
/* istanbul ignore file */
/* tslint:disable */
/* eslint-disable */
export const $PresignedURL = {
properties: {
url: {
type: 'string',
description: `Fully-signed URL — call it with the returned method, no auth header needed.`,
isRequired: true,
},
method: {
type: 'Enum',
isRequired: true,
},
expires_at: {
type: 'string',
isRequired: true,
format: 'date-time',
},
},
} as const;
@@ -7,6 +7,8 @@ import type { BucketObject } from '../models/BucketObject';
import type { BucketObjectListResponse } from '../models/BucketObjectListResponse'; import type { BucketObjectListResponse } from '../models/BucketObjectListResponse';
import type { CopyBucketObjectRequest } from '../models/CopyBucketObjectRequest'; import type { CopyBucketObjectRequest } from '../models/CopyBucketObjectRequest';
import type { CreateBucketRequest } from '../models/CreateBucketRequest'; import type { CreateBucketRequest } from '../models/CreateBucketRequest';
import type { PresignedURL } from '../models/PresignedURL';
import type { PresignObjectRequest } from '../models/PresignObjectRequest';
import type { UpdateBucketObjectRequest } from '../models/UpdateBucketObjectRequest'; import type { UpdateBucketObjectRequest } from '../models/UpdateBucketObjectRequest';
import type { UpdateBucketRequest } from '../models/UpdateBucketRequest'; import type { UpdateBucketRequest } from '../models/UpdateBucketRequest';
import type { CancelablePromise } from '../core/CancelablePromise'; import type { CancelablePromise } from '../core/CancelablePromise';
@@ -131,6 +133,31 @@ export class StorageService {
mediaType: 'application/json', mediaType: 'application/json',
}); });
} }
/**
* Mint a short-lived URL the client calls against object storage directly — keeps large transfers off the backend. Requires BACKEND_STORAGE_DRIVER=s3; the URL's host comes from S3_PUBLIC_ENDPOINT (or S3_ENDPOINT when unset). Signatures bind the host — a presigned URL is valid only at the endpoint it was minted for.
* @returns PresignedURL Presigned URL
* @throws ApiError
*/
public static presignBucketObject({
bucketId,
requestBody,
}: {
bucketId: string,
requestBody: PresignObjectRequest,
}): CancelablePromise<PresignedURL> {
return __request(OpenAPI, {
method: 'POST',
url: '/buckets/{bucketID}/object-presigns',
path: {
'bucketID': bucketId,
},
body: requestBody,
mediaType: 'application/json',
errors: {
404: `Bucket or object not found (download)`,
},
});
}
/** /**
* @returns void * @returns void
* @throws ApiError * @throws ApiError