Skip to content

ADR-012: Bulk PDF upload — hybrid pipeline with on-prem processing agent

Status

In-Review. The architecture, including the multipart reliability refinements and provisional timing defaults, was approved by the product owner on 2026-08-11; that approval is not approval to merge a rollout wave or enable production. Supersedes the unmerged branch document ADR-008-bulk-pdf-s3-only.md from PR #2373 (which collided with main's ADR-008 numbering and was contradicted by that branch's own implementation).

Context

Bulk PDF upload (epic #2223) lets project admins deliver full-text PDFs themselves instead of emailing them to the SyRF team. Final PDF storage is the Edinburgh IIS web server (ecrf1.clinicaltrials.ed.ac.uk/camarades/, ~700 GB of PDFs already served from there), reachable only from inside the university network — nothing in AWS or GKE can write to it.

The feature's storage architecture flip-flopped across 2026: hybrid (S3 staging + on-prem final, discussion #2094, Oct 2025) → S3-only (branch ADR-008, Jan 2026) → hybrid again (decision C-2, Mar 2026, recorded only as a Teams confirmation). The contradiction was never resolved in a merged decision record. This ADR settles it.

Decision

  1. Final storage stays on the Edinburgh IIS share. S3 (syrfapp-uploads*) is staging only; completed ZIPs expire after 7 days. Incomplete multipart uploads have a separate lifecycle cleanup rule as a safety net.
  2. One processing locus, on-prem. A new .NET worker (PDF agent) performs download, ClamAV scanning (clamd companion container), validation, extraction, conflict handling, and the final filesystem write. The same image runs in-cluster for staging/previews (writing a PVC) and as a docker application on the arrnc-api host for production (writing a CIFS mount of the IIS share), managed via server-config.
  3. The Lambda stays thin. The existing s3-notifier gains a BulkPdfUpload kind: it validates metadata, generates a 12 h presigned GET, and sends one MassTransit command to the agent's queue. No scanning or extraction in Lambda.
  4. Domain authority stays in the PM service. The agent reports per-file outcomes; PM performs study matching, marks studies, computes counts, and generates the CSV report in a single-writer consumer.
  5. The browser builds the ZIP; the server owns its multipart upload session. Users select a folder of PDFs; the client validates (PDFs only), previews matches against the search, and zips in a web worker (STORE). It initiates a PM-backed S3 multipart session bound to exact size and whole-object SHA-256, then uploads parts directly to S3 using short-lived presigned UploadPart URLs whose SigV4 signature binds the exact part Content-Length and SHA-256. The browser uploads an exact known-length Blob.slice; its user agent supplies the forbidden Content-Length header, so that header is not returned as a JavaScript-settable RequiredHeaders value. The API does not proxy bytes.
  6. No cloud credentials or inbound connections on university hosts. The agent pulls S3 via presigned URLs and AMQP outbound only.
  7. PM owns upload liveness and completion. A configurable 20-minute lease is renewed by a 60-second heartbeat and after the backend verifies a completed part; part URLs have a separate 5-minute request-start validity. The API also advertises a configurable 10-minute maximum client UploadPart request duration; it is not the URL-expiry window. Only the backend calls CompleteMultipartUpload, after an atomic Uploading → Completing transition and exact manifest verification. That transition atomically persists a durable completion-operation claim shared by the original API request and PM recovery. The claim is three minutes by default, the server-owned storage operation is bounded to two minutes, and every storage side effect and result mutation revalidates exact live ownership. PM excludes a live claim, takes over an expired one atomically, and applies the existing 60-second recovery grace to legacy Completing rows with no claim. A final claimed object HEAD is mandatory before PM may enter Aborting. These accepted defaults remain subject to the pre-launch slow-network/background-device timing gate.
  8. Terminal state converges monotonically. Abandon, cancel, and lease expiry atomically enter Aborting; no heartbeat, stale callback, reconnect, or late successful part may restore Uploading or trigger completion. The API persists the latest issued part-URL expiry before returning that signature. After entering Aborting, PM promptly issues AbortMultipartUpload, but cannot count an absence observation until max(AbortStartedAt, LatestSignedPartUrlExpiresAt) + MaximumUploadPartRequestDuration. With the accepted five-minute request-start validity and ten-minute maximum active request, the worst-case initial quiescence boundary is therefore 15 minutes, not a fixed post-abort grace. PM repeatedly aborts, lists parts, and checks the staging object. It requires two joint empty/NoSuchUpload MPU and object-absence observations at least one 30-second sweep interval apart; any non-empty MPU or present object resets that proof. A present object is deleted and must be re-observed absent on a later pass. Only then may PM record Abandoned/Cancelled and release active capacity. Only legacy jobs that truly have no MPU may terminalize on direct authoritative absence proof. Each cleanup pass first atomically acquires a durable, separate exact-token abort-sweep lease (three-minute default) and is bounded to a shorter storage timeout (two-minute default); active claims are not selected, expired claims are safely taken over, and every storage call and proof mutation revalidates the unexpired claim. Normal passes release promptly; process/host cancellation safely leaves the claim for expiry takeover rather than delaying shutdown. The abort-sweep lease is not reused as the completion-operation lease. Conversely, a winning Completing/Uploaded transition rejects late abandon. Retry v1 creates a fresh independent session; it is not resumability.
  9. Processing requires a PM claim before side effects. After object completion, the agent's first operation is an atomic PM claim of the exact upload/size/hash binding. Only an accepted claim returns the opaque token required on progress and finalize messages; download, scratch, sentinel, scan, and copy all happen after the claim.
  10. The browser renders truthful, reconciled state. Uploading means an UploadPart request or valid bounded retry is actually active. Recoverable loss is shown as retrying; terminal local failure stops animation, queues idempotent abandonment durably, and offers fresh Retry. PM state is reconciled on reconnect/reload through SignalR plus heartbeat/status and bounded polling fallback. A terminal server state cancels local work; a completion winner is rendered as completion.

Consequences

Positive

  • Zero broken links: a study's PDF link activates only after the file is verifiably on disk.
  • One place to debug processing; no cross-cloud two-hop pipeline (the #2373 shape).
  • No new AWS infrastructure beyond a chart-declared lifecycle rule; no clamd-in-cloud.
  • Preview/staging environments run the identical pipeline in-cluster, so the feature is E2E-testable per PR.
  • ClamAV signatures stay current via freshclam without image rebuilds.
  • A killed browser, powered-off device, or severed connection cannot leave an upload Pending on the assumption that the original non-resumable transfer may spontaneously continue.

Negative / accepted

  • Production depends on a single manually-applied host (arrnc-api) and a CIFS mount; mitigated by fail-closed sentinel checks, durable queueing while the agent is down, and server-config's recovery/verify roles.
  • Unscanned bytes touch the agent's isolated container before scanning: the ZIP is downloaded and extracted into container-scratch space, and every extracted file is scanned there before anything is copied to the share or any served location. The security boundary is "nothing unscanned ever reaches final storage", not "nothing is extracted pre-scan" — scratch is private to the agent container (same trust boundary as scanning inside a Lambda).
  • AMQP to rabbitmq.camarades.net:5672 is currently plaintext over the internet, and this applies to both out-of-cluster clients: the existing production Lambda and the new PDF agent on arrnc-api. A separate hardening PR adds AMQPS; both flip their URIs when it lands.
  • The bulk layout (projects/{p}/searches/{s}/pdfs/…) coexists with the legacy flat layout; link generation branches on Study.BulkPdfDeliveredAt.
  • Multipart upload introduces a PM sweeper and state/storage reconciliation path. Capacity is deliberately retained through the derived latest-signed-URL quiescence boundary and two-observation cleanup proof; the storage lifecycle rule is only a fallback, so transient S3 errors do not manufacture a terminal state.
  • The accepted timing defaults trade prompt interruption detection against slow links and browser throttling. Staging must validate slow 1 GB upload, prove each exact 16 MiB part completes safely inside the advertised 600-second request timeout, and exercise transient loss, heartbeat jitter, and device sleep/background behaviour before production enablement; unsafe evidence requires an explicit narrow timing decision rather than silent retuning.

Alternatives considered

  • S3-only storage, no agent (branch ADR-008; revisited 2026-08-11 as "GKE worker + S3 final"): cleanest platform-native end-state, but changes the PDF serving model and contradicts the standing storage decision; rejected consciously.
  • Keep #2373's two-hop pipeline (scan-in-Lambda → safe bucket → agent): requires clamd reachable from AWS (new always-on infra), doubles S3 I/O, and splits processing across a cloud/on-prem boundary; dominated once an on-prem writer exists anyway.
  • Direct upload to an ingest service on arrnc-api (no S3): loses durable staging, presigned upload UX, and per-PR preview isolation; couples availability to one host.
  • One presigned PUT plus read-time/24-hour expiry: the server cannot know promptly that the browser's non-resumable request is irrecoverably severed, cannot make continuation impossible, and may strand active capacity. Rejected in favour of renewable server-owned multipart state.
  • Browser-owned multipart completion: allows a stale client to race terminal abandonment and publish an object after PM has ended the session. Rejected; completion is backend-only.
  • Resume/adopt prior parts on Retry: adds cross-session trust and ownership complexity. Rejected for v1; Retry is explicit and fresh after authoritative cleanup.

References

Design: docs/superpowers/plans/2026-08-11-bulk-pdf-upload-v2-design.md · Plan: docs/superpowers/plans/2026-08-11-bulk-pdf-upload-v2-plan.md · Epic #2223 · discussions #2093/#2094/#2095 · superseded attempt PR #2373 · ADR-010 (s3-notifier IAM via ACK) and its 2026-08-11 audit.