Bulk PDF Upload v2 — Design¶

Companion implementation plan: 2026-08-11-bulk-pdf-upload-v2-plan.md. Decision record: ADR-012.

1. Context¶

Project admins cannot upload full-text PDFs themselves; they email them to the SyRF team, who copy files manually onto the Edinburgh web server (https://ecrf1.clinicaltrials.ed.ac.uk/camarades/, ~700 GB of existing PDFs). Epic #2223 (open since 2020) asks for self-service bulk upload with zero broken links, validation reporting, and safe archive handling. Spec/plan/backlog: discussions #2093 / #2094 / #2095.

Two prior implementation attempts did not land:

#2298 (Jan 2026) — closed unmerged.
#2373 (Mar–May 2026) — a functionally complete pipeline, but with no solution entries, no CI build, no chart, no feature flag, no production deploy path for its two new services, an ADR contradicting the implemented architecture, 36 unresolved review threads, and failing quality gates. It is ~4 months behind main.

v2 is a fresh implementation. #2373 is used as reference material only — its domain-model patterns, guard logic, and test cases inform this design; its code is not merged. #2373 will be closed as superseded when this plan merges.

2. Architecture overview¶

flowchart LR
    subgraph Browser
        A[Folder picker /<br/>drag-and-drop] --> B[Client validation<br/>PDFs only]
        B --> C[Pre-upload match preview<br/>vs search PdfRelativePaths]
        C --> D[Client-side ZIP<br/>web worker, STORE]
        D --> U[Multipart upload client<br/>checksums + heartbeat]
    end
    U -- initiate / sign part /<br/>ack part / complete --> API[SyRF API + PM domain<br/>server-owned session]
    U -- presigned UploadPart --> S3[(S3 syrfapp-uploads*<br/>Projects/_bulk-staging/)]
    API -- backend-only<br/>CompleteMultipartUpload --> S3
    API -- abort + ListParts<br/>grace + two-empty proof --> S3
    S3 -- ObjectCreated --> L[s3-notifier Lambda<br/>uploadkind=BulkPdfUpload]
    L -- IProcessBulkPdfUploadCommand<br/>+ presigned GET --> MQ[(RabbitMQ)]
    MQ --> AG[PDF agent<br/>.NET worker]
    AG -- INSTREAM scan --> CL[clamd sidecar]
    AG -- extract + copy --> ST[(Final storage<br/>prod: ecrf1 CIFS<br/>staging/preview: PVC)]
    AG -- IFinalizeBulkPdfUploadCommand --> MQ2[(RabbitMQ)]
    MQ2 --> PM[PM single-writer consumer<br/>match, mark studies,<br/>counts, CSV report]
    PM -- SignalR via Project stream --> Browser

Key properties:

The API never proxies the bytes, but owns the upload session. The browser sends exact known-length parts directly to S3 using short-lived presigned UploadPart URLs whose SigV4 signature binds both content length and checksum. The API initiates the multipart upload, verifies each completed part against S3, renews the lease, performs the only accepted CompleteMultipartUpload, and drives authoritative abort cleanup.
Disconnected clients converge promptly. Uploading has a renewable server lease; a client that can no longer continue either sends an idempotent abandon command or is caught by the lease sweeper. A terminal browser state never waits for the browser to successfully report its own failure.
One processing locus. All scanning, validation, extraction, and copying happen in the agent. The Lambda is a thin notifier (extension of the existing s3-notifier).
Domain authority stays in PM. The agent is a "dumb pipe": it never matches files to studies and holds no cloud credentials. Matching, study marking, counts, and the CSV report are computed by the PM service from the agent's reported per-file outcomes.
Outbound-only from the university network. The agent pulls from S3 via a presigned GET and talks AMQP outbound. Nothing connects inbound to UoE hosts.

3. Resolved decisions¶

The full decision register (batch-grill, 2026-08-11). IDs are referenced throughout.

ID	Decision
A1	Per-search upload from the Systematic Searches page; effective payload cap 1 GB
A2	Matching = normalized relative path vs `Study.PdfRelativePath`; normalizer test vectors ported from #2373's `PdfPathNormalizerTests`
A3	All three conflict policies: `skip-existing` (default), `replace-always`, `replace-if-differs` (SHA-256)
A4	Folder-based upload: user selects/drops a folder of PDFs; client validates, previews matches, and builds the ZIP itself (web worker, STORE). ZIP is a transport detail
A5	Feature flag `bulkPdfUpload` (env-mapping generator); off in previews/production until rollout
A6	Per-file outcome taxonomy: Copied / Skipped / Replaced / Unmatched / Missing / Invalid / Infected. UI summary + downloadable CSV report
A7	Dedicated `BulkPdfUpload` authorization policy, initially granted to project-design roles
B1	`BulkPdfUploadJob` embedded in the Project aggregate (SearchImportJob precedent); absolute-snapshot counts; single terminal-writer consumer; `BsonClassMap` registered
B2	Server-owned S3 multipart session; ZIP key under `Projects/_bulk-staging/`; exact-content-length + checksum-bound presigned `UploadPart` URLs with 5-minute request-start validity; backend-only completion
B3	PM is authoritative; state changes use the existing Project SignalR stream with heartbeat/status and bounded polling reconciliation because real-time delivery can be missed
B4	Files land at `projects/{projectId}/searches/{searchId}/pdfs/{entry-path}`; `PdfRelativePath` is never rewritten — link generation composes the prefix for bulk-delivered PDFs
B5	`PdfBaseUrl` env-supplied config (no hardcoded Edinburgh URL)
C1	Notifier dispatch rewritten to `Enum.TryParse` + `switch`, unknown kinds = logged no-op (also fixes live single-study-PDF Lambda errors)
C2	Point-to-point `Send` of `IProcessBulkPdfUploadCommand` (one owner), not publish
C3	Agent reads S3 via 12 h presigned GET generated by the Lambda; no AWS credentials on the agent
C4	Completed staging ZIPs expire via a 7-day prefix lifecycle rule; incomplete MPUs have a separate lifecycle cleanup safety net. Both are declared in the ACK Bucket CR and neither substitutes for server state convergence
C5	Notifier ships via the standard zip-version promotion
D1	New .NET worker `src/services/pdf-agent/` with slnf, GitVersion, Docker image, Helm chart
D2	Extract with guards, then per-file clamd INSTREAM scan + `%PDF` magic check; any infection ⇒ whole upload `Infected`, nothing copied
D3	Batched absolute-snapshot progress + exactly one terminal finalize (always sent); idempotent under redelivery
D4	Same agent image in every environment: in-cluster (chart) for staging + previews writing a PVC; docker app on arrnc-api for production writing CIFS
D5	RabbitMQ over the existing `rabbitmq.camarades.net:5672`; AMQPS is a separate, non-gating hardening PR
E1	Production agent is gatekeeper-deployed on arrnc-api (mission-control-migration-worker precedent)
E2	Single bind volume `/srv/data/syrf-pdf/production` containing `scratch/` (ext4) and `output/` (nested CIFS mountpoint); sentinel-file guard before writes
E3	clamd as an Ansible-managed companion container (mailpit precedent) on a new non-internal `syrf-pdf-net` bridge; reached as `clamd:3310`; freshclam keeps signatures current
E4	CIFS mount via the existing `cifs_mounts` role + a new `cifs/<name>` age secret scope; `mount_user` uid = agent `container_uid` from day one
E5	New `github_runner` registration for `camaradesuk/syrf` on arrnc-api; GHCR pulls via ephemeral `GITHUB_TOKEN`; container secrets via workflow `-e` from GitHub environment secrets
F1	Delivery as waves of small PRs (see plan doc); ADR-010 Phase-5 unblock is a bundled prerequisite wave
F2	#2373 closed as superseded at plan merge; branch kept for reference
F3	ADR-012 records the architecture; stale docs updated in the PRs that touch them
G1	Upload lease defaults are configurable 20 minutes / 60-second heartbeat; a server-verified completed part also renews the lease. Values remain subject to slow-1-GB/network-sleep validation before launch
G2	Upload state is monotonic: completion and abort race through optimistic atomic transitions. `Aborting`/`Abandoned`/`Cancelled` can never return to `Uploading`; a winning `Completing`/`Uploaded` rejects late abandon
G3	Cleanup is authoritative and cross-replica safe: the latest issued part-URL expiry is persisted before its signature is returned; cleanup proof cannot start before `max(AbortStartedAt, LatestSignedPartUrlExpiresAt) + MaximumUploadPartRequestDuration`. Each bounded storage pass holds a separate exact-token durable abort claim, calls abort/ListParts/HEAD, and requires two joint empty/`NoSuchUpload` plus object-absence observations at least 30 seconds apart. Nonempty parts or an object reset proof; an object is deleted and must be re-observed absent. Normal passes release promptly, while crash/host cancellation recovers through claim expiry. S3 lifecycle is only a safety net
G4	`Abandoned` and `Cancelled` release active capacity only after storage cleanup is verified. Retry v1 always creates a fresh independent upload/session; no resumability is claimed
G5	The PDF agent must claim the successfully completed upload in PM before download or any other side effect; the claim token binds progress/finalize messages
G6	API advertises a configurable 600-second maximum client `UploadPart` request duration, distinct from the 300-second presigned request-start validity; staging must validate the timing before launch
G7	API completion and PM recovery share one durable exact-token completion-operation lease (provisional 180 seconds) around a shorter server-owned storage timeout (120 seconds). PM excludes live claims, atomically takes over expired claims, applies the 60-second grace only to legacy missing-claim rows, revalidates before every storage effect/write, and performs a final claimed HEAD before Aborting. The abort-sweep claim is separate

4. Component design¶

4.1 Domain (Project Management core)¶

New embedded entity on the Project aggregate (pattern: SearchImportJob):

BulkPdfUploadJob
  Id                 Guid (uploadId; client-visible)
  SystematicSearchId Guid
  Status             Uploading | Completing | Uploaded | Scanning | Copying | Finalizing
                     | Aborting | Abandoned | Cancelled | Complete | Failed | Infected
                     | [legacy BSON only: Queued | Expired]
  ConflictPolicy     SkipExisting | ReplaceAlways | ReplaceIfDiffers
  StorageBinding     { key, multipartUploadId, virtualHost }
  PayloadBinding     { expectedSize, expectedSha256, partSize, partCount }
  VerifiedParts      bounded { number, size, checksumSha256, verifiedAt }[]
  UploadLease        { lastHeartbeatAt, leaseExpiresAt }
  LatestSignedPartUrlExpiresAt DateTime?
  CompletionClaim    { opaque token, expiresAt } (internal, durable, never client-visible)
  StateRevision      long
  ProcessingClaim    { claimedAt, opaque claimToken }
  AbortState         { disposition, startedAt, lastAttemptAt, attemptCount }
  AbortSweepClaim    { opaque token, expiresAt } (internal, durable, never client-visible)
  RetryOfUploadId    Guid?
  Counts             { Total, Copied, Skipped, Replaced, Unmatched, Missing, Invalid }
                     (Missing is null until finalize — it is only computable once the whole
                      ZIP has been processed and compared against the search's study paths;
                      progress snapshots never set it and the UI shows "—" until completion)
  InfectedFiles      string[]   (bounded: first 100)
  FailureReason      string?
  ReportFileKey      string?    (S3 key of the CSV report)
  CreatedAt / CompletedAt / CreatedBy

Rules:

The server is authoritative and status transitions are idempotent and monotonic. Optimistic compare-and-save persists Uploading → Completing or Uploading → Aborting before the corresponding S3 side effect. Whichever transition wins rejects the loser. Once Aborting, Abandoned, or Cancelled wins, no heartbeat, part signing, part acknowledgement, stale callback, or reconnect can restore Uploading.
Uploading carries a configurable lease (accepted default: 20 minutes) and clients send heartbeats at the server-advertised interval (default: 60 seconds). A part acknowledgement renews only after the backend has listed S3 and verified that part's number, exact size, ETag, and checksum. Presigned UploadPart URLs bind the expected byte length and SHA-256, have a separate 5-minute request-start-validity window, and never extend the session lease. The API advertises a separate configurable 600-second maximum duration for an already-started client part request.
Completion is backend-only. The backend first verifies the complete consecutive part manifest, atomically enters Completing with one durable completion-operation claim, calls S3, and reconciles ambiguous results with authoritative object HEAD identity metadata/content type/size/ETag. The original API operation and PM recovery use that same claim protocol: a live claim excludes recovery, an expired claim has one atomic takeover winner, and legacy missing-claim rows become eligible only after the 60-second recovery grace. Before every HEAD, ListParts, CompleteMultipartUpload, final HEAD, or proof mutation the exact owner is revalidated. The provisional 180-second claim outlives the 120-second server-owned storage timeout. A final claimed HEAD is required before recovery may enter Aborting; the abort sweeper uses a separate claim. The agent later verifies the downloaded ZIP's actual whole-object SHA-256 before processing. Clients can request completion but never call S3 CompleteMultipartUpload themselves.
A signed part URL is never returned until its expiry has been persisted on the job. Abandon, cancel, or expiry first enters nonterminal Aborting, which still consumes active-upload capacity. For an MPU, PM promptly issues AbortMultipartUpload; the sweeper repeats abort, ListParts, and object HEAD, but cannot count cleanup proof before max(AbortStartedAt, LatestSignedPartUrlExpiresAt) + MaximumUploadPartRequestDuration. With the accepted 300-second URL request-start window and 600-second active-request maximum this is at most 15 minutes after an immediate abort. Terminalization requires two joint empty or NoSuchUpload MPU and object-absence observations at least one 30-second sweep interval apart; seeing any part or present object resets the proof. A present object is deleted and must be re-observed absent on a later pass. Only then does PM record Abandoned or Cancelled and release capacity/deletion guards. Only legacy jobs that truly have no MPU may terminalize on direct authoritative absence proof. The bucket's incomplete-MPU lifecycle rule is a final cleanup safety net, not proof of domain completion.
Heartbeat, part, and completion responses include the authoritative session snapshot. Heartbeat/renewal, part signing, and part acknowledgement against a terminal session are rejected with a typed problem response; idempotent status/completion reconciliation returns the authoritative winning outcome. Terminal states are never hidden by a generic active success. Processing progress snapshots remain absolute and monotonic (a stale snapshot never regresses counts).
Persisted upload-state changes flow through the existing Project SignalR publication path where appropriate. Clients still reconcile with heartbeat/status and bounded polling because a real-time notification can be delayed or missed.
Retry v1 is deliberately non-resumable. A user retries only from an eligible terminal job and receives a fresh upload ID, S3 multipart upload ID, lease, part set, and payload binding, linked by RetryOfUploadId; old parts are never adopted.
Per-file outcomes are not stored in Mongo (unbounded); they live in the CSV report on S3. Mongo stores counts + bounded infected list.
Legacy Queued/Expired values remain readable for BSON compatibility, but the multipart protocol never infers expiry on read. A hosted sweeper queries due sessions and persists all abandonment and cleanup transitions.
History is bounded: on new-job creation, terminal jobs beyond the most recent 20 per search are pruned from the aggregate in the same save (their CSV reports persist in S3 independently). Prevents unbounded Project-document growth toward the 16 MB BSON limit.
SignalR delivery is explicit wiring, not free: ProjectDetailsDto enumerates job collections explicitly (SearchImportJobs, BulkStudyUpdateJobs, RiskOfBiasJobs) and AutoMapper drops unmapped properties — PR-1 must add BulkPdfUploadJobDto, the BulkPdfUploadJobs property, and the mapping, or the progress UI stays dark.
BsonClassMap.RegisterClassMap entries for BulkPdfUploadJob and its value objects with UnmapProperty(x => x.Project), matching every other embedded entity in ProjectRepository.cs (#2373 omitted this — a runtime serialization crash risk).

Study side:

Study.BulkPdfDeliveredAt : DateTime? and Study.BulkPdfDeliveredPath : string? — set together when a bulk upload actually delivers this study's PDF. The delivered path is the normalized path the agent really wrote, stored immutably: a later ordinary study update that rewrites PdfRelativePath (e.g. StudyReferenceFileParser, bulk study update) can then never break the link — the URL keeps serving the file that exists.
Link generation: when delivered, the PDF URL is {PdfBaseUrl}projects/{projectId}/searches/{searchId}/pdfs/{BulkPdfDeliveredPath} with each path segment percent-encoded (preserving /) so filenames containing #, ?, or spaces produce working links; otherwise the legacy {PdfBaseUrl}{PdfRelativePath}. Legacy studies are untouched (D-B4), including the existing absolute-URL special case: today's Study.GetLinkToPdf returns http-prefixed PdfRelativePath values unchanged, and the centralized builder must preserve exactly that behaviour or working legacy links corrupt.
One URL builder, all call sites: the repository currently has five independent PDF-URL constructions that bypass Study.GetLinkToPdf — StudyROBDto, StudyListItemDto, StatsWithIncompleteDto/StudyBaseDto, StudyDto, and PdfConverterService (used for risk-of-bias conversion). PR-1 centralizes URL construction in one bulk-aware helper and routes every one of those call sites through it; leaving any of them out ships broken links or failed ROB conversion for bulk-delivered studies.

4.2 Path normalization contract (correctness backbone)¶

One rule set, three consumers: client pre-upload matching (TS), link generation (C# and TS), agent/PM matching (C#).

Rules: Unicode NFC → fold typographic punctuation to ASCII → invariant casefold → \ → / → collapse repeated / → trim → strip leading ./ and /.

The punctuation fold (curly single/double quotes → '/", en/em dash → -, non-breaking space → space) is not tidiness: a study's PdfRelativePath is routinely imported from a CSV that has been through Word or Excel, whose auto-correct rewrites straight quotes as curly ones and hyphens as en-dashes, while the PDF on the uploader's disk keeps the plain ASCII name. Without the fold those two never compare equal, so the study is reported Missing and its file Unmatched — the file lands on the share but no study links to it. This is the mismatch #2373 was fixed to address, and the live use-case CSV is exactly that kind of data. The mapping is deliberately narrow: only characters with an unambiguous ASCII counterpart that auto-correct actually produces (an ellipsis or bullet is left alone, since folding it would invent a match rather than recover one).

Two companion contracts ride with normalization:

Portability contract (the final storage is a Windows-backed IIS/CIFS share): entries containing Windows-reserved characters (< > : " | ? *, control chars), reserved device names (CON, PRN, AUX, NUL, COM1-9, LPT1-9), trailing dots/spaces on any component, or components longer than 240 UTF-8 bytes are rejected as Invalid — enforced in the client (friendly pre-upload error) and re-enforced in the agent (authoritative).
Collision contract: the normalized-key map is built up front; if two entries collapse to the same canonical key (A.pdf vs a.pdf from a case-sensitive filesystem), the upload is rejected before any write — otherwise ZIP order would silently decide which file wins. Client checks first; agent re-checks authoritatively.
Root-relative contract: the browser strips the selected folder's own name (folder-picker webkitRelativePath includes it), so ZIP entry paths are relative to the inside of the chosen folder for picker, drag-and-drop, preview, and matching alike.

The canonical test-vector file src/libs/kernel/SyRF.SharedKernel/PdfPathNormalization/normalization-vectors.json is loaded by both the C# and TS test suites; a vector added on one side fails the other side's build if unimplemented. Vectors seed from #2373's PdfPathNormalizerTests (252 lines) plus PDFs (2).csv real-world names.

4.3 API endpoints (flag-gated, `BulkPdfUpload` policy)¶

Endpoint	Purpose
`POST api/projects/{p}/searches/{s}/bulkPdfUpload/uploads`	Idempotently initiate `Uploading`; bind upload ID to exact size/SHA-256/part geometry and a server-owned S3 MPU. One active-capacity job per search
`GET .../bulkPdfUpload/uploads/{uploadId}`	Authoritative session snapshot used on load/reconnect and polling fallback
`POST .../uploads/{uploadId}/parts/{part}/signature`	Return an exact-content-length + checksum-bound short-lived `UploadPart` URL only for a live `Uploading` session; the session snapshot advertises the independent client request timeout
`POST .../uploads/{uploadId}/parts/{part}/completed`	Verify the part through S3, persist its binding, and renew the lease at this completed-part boundary
`POST .../uploads/{uploadId}/heartbeat`	Renew a still-live lease and return authoritative state plus the advertised heartbeat interval; terminal sessions receive typed rejection
`POST .../uploads/{uploadId}/complete`	Verify all parts, atomically claim `Completing`, perform backend-only completion, and reconcile the completed object
`POST .../uploads/{uploadId}/abandon`	Idempotent command-ID-based transition to `Aborting(Abandoned)`; cleanup continues asynchronously if the first abort is inconclusive
`POST .../uploads/{uploadId}/cancel`	Explicit user cancellation through `Aborting(Cancelled)`; late cancellation cannot override completion/processing
`POST .../uploads/{oldUploadId}/retry`	Create a fresh independent session linked to the eligible terminal source; no part resume/adoption in v1
`GET api/projects/{p}/searches/{s}/bulkPdfUpload/history`	Upload history for the search
`GET api/projects/{p}/searches/{s}/bulkPdfUpload/{uploadId}/report`	Streams the CSV report from S3
`GET api/projects/{p}/searches/{s}/pdfPaths`	Study `PdfRelativePath` list for the client-side pre-upload match preview. Returns only studies with a non-null `PdfRelativePath` (all the preview needs), keeping the response lean; expected worst case a few MB for very large searches — no pagination

The completed object's metadata includes project/search/upload identity, conflict policy, virtual host, upload kind, and the expected whole-object size/SHA-256. It stays below S3's 2 KB cap: no file manifest travels in metadata (the job document and report carry state). Storage keys and multipart upload IDs never leave the server-facing DTO.

PR-1 owns these server/domain semantics, including the sweeper and claim contract. PR-4 owns the browser implementation and reconciliation UX described in §4.7. Bucket CORS/IAM and the incomplete-MPU lifecycle safety net are deployment concerns verified in the later environment PRs; browser-PUT verification must prove the user-agent-supplied signed Content-Length and JavaScript-set checksum header work from every SyRF origin. The client protocol must not depend on lifecycle timing.

4.4 Notifier (s3-notifier extension)¶

Dispatch: Enum.TryParse<UploadKind> + switch; default: logs and returns (no throw, no retry). This also stops the current production behaviour where objects without uploadkind/virtualhost metadata (today's single-study PDF uploads) make the Lambda throw (S3FileReceivedFunction.cs:107-142).
case BulkPdfUpload: validate metadata; generate 12 h presigned GET for the object. The Lambda role already holds s3:GetObject in every environment — via the ACK chart for staging/previews (iam-role.yaml:88-89) and via Terraform for production until ADR-010 Phase 6 (camarades-infrastructure/terraform/lambda/main.tf); Send IProcessBulkPdfUploadCommand { ProjectId, SearchId, UploadId, ConflictPolicy, ExpectedSizeBytes, ExpectedSha256, ZipDownloadUrl, DateTimeEventOccurred } to the agent queue on the vhost from metadata (preview routing works unchanged). The agent uses the expected size/hash only after PM confirms that they match the persisted, completed session.
Contract lives in SyRF.ProjectManagement.Messages (the notifier already depends on it).

4.5 Agent (`src/services/pdf-agent/`, new)¶

.NET worker, MassTransit consumer, concurrency 1. Per job:

Claim before side effects: request IClaimBulkPdfUploadProcessingCommand from PM before checking storage, creating scratch state, downloading, scanning, or writing. PM accepts only the exact project/search/upload/size/hash binding in Uploaded, atomically persists a claim, and returns an opaque token. Redelivery of the same binding is idempotent; a wrong binding or terminal/aborting session is rejected without disclosing the token. Every progress/finalize command carries the token and PM rejects missing or mismatched tokens. This closes the race where a notifier event is already in flight when cancel/abort wins.
Sentinel check: output root must contain the marker file .syrf-storage-ok. Missing marker ⇒ fail closed with StorageUnavailable (protects against writing "under" an absent CIFS mount — see §6.3). Provisioning: server-config creates it on the production share (mount-guarded task); the agent chart creates it on staging/preview PVCs via an init container — a fresh PVC must not fail-closed forever.
Download the ZIP via the presigned URL, streaming to scratch/{uploadId}/upload.zip with a byte counter that aborts past the 1 GB cap (client enforcement is advisory; a modified client could sign any size) — over-cap ⇒ terminal Failed. An expired URL (agent down long enough that a redelivered message outlives the 12 h presign) returns 403 — this maps to a terminal Failed status with a clear reason, not a retry loop; the user re-uploads.
Extract entry-by-entry with guards: Zip-Slip/path-traversal rejection, __MACOSX/ and ._*/.DS_Store skip (defence in depth — our client builds the ZIP), zip-bomb caps (≤ 10,000 entries, ≤ 4 GB total uncompressed, ≤ 500 MB per entry), the portability and collision contracts from §4.2, entry paths capped at 512 UTF-8 bytes (longer ⇒ Invalid), and only .pdf extensions (case-insensitive) — anything else is Invalid regardless of content, so a crafted payload.html starting with %PDF can never land under the IIS document root with an actively-served extension.
Scan each extracted file via clamd INSTREAM (clamd:3310). clamd is deployed with StreamMaxLength/MaxScanSize/MaxFileSize raised to cover the 500 MB entry cap (the image defaults are far lower and would silently truncate or reject) — a scan error or limit hit is a terminal Failed, never an unscanned pass. Any hit ⇒ record infected names, delete scratch, finalize as Infected — nothing is copied (all-or-nothing, D-D2).
Validate %PDF magic bytes per file (non-PDF ⇒ per-file Invalid outcome).
Copy to output/projects/{p}/searches/{s}/pdfs/{normalized-entry-path} via temp-file + atomic rename, honouring the conflict policy (ReplaceIfDiffers = SHA-256 compare against the existing file).
Report: IReportBulkPdfUploadProgressCommand (absolute snapshot) every 50 files; exactly one IFinalizeBulkPdfUploadCommand carrying status + per-file outcome entries, sent in a finally so a crash still terminates the job as Failed. Size budget: entries are { path ≤ 512 B, outcome, detail ≤ 256 B } (study identity is attached later by PM, which owns matching) — the caps bound the message at ≈ 8 MB absolute worst case, ~2 MB typical; well inside broker limits, and the caps exist precisely to keep that bound.
Outcome journal (crash-safe idempotency), two-phase: before each entry's atomic rename, the agent appends a pending record ({path, plannedOutcome}) to scratch/{uploadId}/journal.jsonl; immediately after the rename returns, it appends the matching committed record. A single-phase journal (write only after, or only before, the rename) leaves a crash window in one direction or the other; two phases plus a reconciliation rule close both. Redelivery replays the journal: a path with a committed record reuses that recorded outcome outright (no recompute — a file copied pre-crash stays Copied, never degrades to Skipped under skip-existing); a path with only a pending record is reconciled against the filesystem — if the destination file's hash matches the still-available scratch copy, the rename evidently completed (upgrade to committed and reuse the outcome); otherwise the rename never completed and the copy is redone. Counts/CSV stay truthful either way.
Cleanup scratch. The startup orphan sweep only removes upload directories with no active-lock file and an mtime older than 24 h — never a directory another live instance may own (in-cluster rolling updates can briefly run two pods against one PVC; the chart additionally uses the Recreate deployment strategy to avoid overlap by construction).

The agent holds: RabbitMQ credentials (workflow-injected secret) and a filesystem path. It has no AWS keys, no Mongo access, no HTTP surface, no inbound ports.

4.6 PM consumers (single writer)¶

One receive endpoint, concurrency 1, hosting both consumers — progress and finalize are serialized, eliminating aggregate write races (the lesson #2373 learned late):

Progress: monotonic snapshot update of Counts on the job.
Finalize: match reported file paths against IStudyRepository.GetPdfPathsBySearchId (normalized both sides); set BulkPdfDeliveredAt/BulkPdfDeliveredPath only for studies whose file outcome actually delivered bytes — Copied, Replaced, or verified-existing Skipped; Invalid and Failed outcomes leave the study untouched, and an Infected finalization marks nothing (all-or-nothing) — the zero-broken-links invariant lives or dies on this qualification. Compute Unmatched (file, no study) and Missing (study, no file); generate the CSV report and store it via IFileService to {prefix}Projects/{p}/Bulk PDF Uploads/reports/{uploadId}.csv; write terminal state + counts + report key in one aggregate save. Idempotent via the job-status guard.
The feature flag gates the API surface only; PM consumers and the agent always process, so in-flight jobs reach terminal state and the flag-off infra smoke works. A flag-gated consumer would strand a job mid-flight — which in turn deadlocks the deletion guard below, since that refuses while any job is non-terminal.
Cross-replica safety: endpoint concurrency 1 is per process, and a rolling PM deploy briefly runs two pods against the shared queue. The endpoint therefore carries a retry policy for Mongo optimistic-concurrency conflicts (the idempotent guards + monotonic merge make retries safe), so a losing concurrent write re-runs instead of dead-lettering a finalize and stranding the job nonterminal.
Deletion guard: a systematic search (or its project) with a non-terminal bulk upload job cannot be deleted — the domain rejects it. Otherwise the agent would keep writing to a deterministic path whose aggregate no longer exists, stranding files and an unprocessable finalize.

Once PR-1's BulkPdfUploadJobDto + ProjectDetailsDto.BulkPdfUploadJobs + AutoMapper mapping are in place (§4.1), subsequent Project saves stream to subscribed clients automatically — no additional plumbing per consumer needed.

4.7 Frontend¶

Angular standalone dialog + @ngrx/signals store on the Systematic Searches page (flag-gated menu action):

Folder selection: drag-and-drop (webkitGetAsEntry traversal) and folder picker (webkitdirectory).
Validation: only directories and PDFs (extension + first-bytes %PDF sniff); offenders listed and upload blocked. OS junk (.DS_Store, __MACOSX/, ._*, Thumbs.db) is silently skipped, never an offender — a valid folder picked on macOS must not be rejected for artifacts the agent's own skip list would ignore anyway.
Pre-upload preview: matched / unmatched / missing table computed client-side against GET .../pdfPaths using the shared normalization rules; user confirms with eyes open.
ZIP built in a web worker with a streaming zip library, STORE mode (PDFs are already compressed; avoids CPU and memory blowup at 1 GB), written to a Blob (browsers spill large Blobs to disk). The whole-payload SHA-256 is computed incrementally from blob.stream(); the ZIP must never be materialised via arrayBuffer(). Peak memory stays at chunk size, not payload size.
Initiate the server-owned multipart session with the exact payload size/SHA-256, split the Blob using the server-advertised part size, and upload each exact known-length Blob.slice. Calculate the part SHA-256 and request a URL whose SigV4 signature binds that checksum and exact Content-Length; JavaScript sets the returned checksum header, while the browser owns the forbidden Content-Length header (it is deliberately absent from RequiredHeaders). Enforce the server-advertised 600-second maximum request duration with the upload transport's timeout/abort support (XMLHttpRequest.timeout/abort(), or AbortController where supported), then acknowledge the part to PM. Only that verified acknowledgement preserves progress and renews the lease. Uploading is shown only while an UploadPart request or a bounded automatic retry is genuinely active.
Send heartbeat at the server-advertised interval and reconcile every response's authoritative state. A recoverable network failure shows “Connection lost — retrying” with progress preserved while a bounded retry can still use the live session. If recovery is no longer possible or the client declares the attempt terminal, abort local requests with AbortController, stop progress/retry loops, discard queued completion, show “Upload interrupted” / “Cancelling”, and offer a fresh Retry.
The terminal browser transition immediately queues an idempotent abandon command. If offline, persist it client-side and resend on reconnect/reload. On restoration, query PM before rendering; when local terminal state disagrees with server Uploading, show “confirming interruption with server”, resend abandon, and never fake active progress. If PM reports Completing/Uploaded, render that winning state instead.
Consume authoritative transitions through the existing Project SignalR stream (selectSignal per repo modernisation rules), with heartbeat/status and bounded polling fallback for missed real-time messages. A server terminal state cancels local in-flight work where possible and ends all retries. An already-issued part may still return success after abort, but neither the browser nor backend may complete a terminal session.
Result summary panel (headline counts, infected/unmatched highlights), history table, CSV report download, and a fresh Retry action. Status text is accessible and describes the true transfer/server state rather than a perpetual animation.

Focused PR-4 tests cover: network drop followed by successful same-session part retry; terminal drop then reconnect; reload with queued abandonment; lost client that never returns (lease sweeper); server abort while the client believes it is active; missed SignalR recovered by heartbeat/poll; late successful part after abort; stale heartbeat cannot resurrect a terminal session; request-timeout cancellation without confusing it with URL expiry; exact Blob.slice part sizing; and both completion-vs-abort winners converging in UI and server state.

4.8 CSV report columns¶

file_path, normalized_path, outcome, study_id, study_title, detail — one row per ZIP entry, plus one row per Missing study (outcome missing). Header row + UTF-8 BOM for Excel friendliness. RFC 4180 quoting throughout, and formula neutralization: any cell beginning with =, +, -, or @ is prefixed with ' — ZIP paths and study titles are user-controlled and the report is opened in Excel by other admins. Covered by CSV tests.

5. Environments¶

	Production	Staging	Preview (per PR)
Bucket	`syrfapp-uploads`	`syrfapp-uploads-staging`	`syrfapp-uploads-pr-{n}`
Notifier	shared Lambda (per ADR-010 state)	staging Lambda (ACK)	per-PR Lambda (ACK)
RabbitMQ vhost	production	staging	per-PR
Agent hosting	docker app on arrnc-api (gatekeeper)	in-cluster (chart)	in-cluster (chart)
Final storage	ecrf1 CIFS mount	PVC	PVC
PDF serving	existing IIS site	static server in agent chart	static server in agent chart
Flag	off until rollout complete	on for rehearsal	per-PR `#preview-config`

Why PVC rather than ecrf1 subfolders for staging/previews: (1) in-cluster agents cannot reach the share — the UoE firewall allows no inbound SMB from the internet, which is the very reason the production agent lives on arrnc-api; (2) per-PR agents cannot live on arrnc-api — its container slots are statically declared in server-config with manual production applies, while previews are created/destroyed automatically per PR; (3) lifecycle isolation — preview PDFs should die with the preview namespace (PVC deletion is automatic; ecrf1 subfolders would orphan test files on the backed-up production estate); (4) security — preview/staging environments must not hold production file-server credentials.

Why not host the staging/preview agents on arrnc-api as gatekeeper docker apps: previews are structurally impossible there — the gatekeeper deploys only pre-declared, manually-applied whitelist slots (its security model), and a preview must run the PR's own agent image to E2E-test agent changes, which a static shared container never could. Staging is feasible but a bad trade: it could not rehearse the real CIFS write anyway (staging data must not touch the production share, so it would write local disk — no more production-like than a PVC), while it would add a second bespoke deploy lane outside the standard cluster-gitops promotion, new serving infrastructure on the pet host for staging's PdfBaseUrl, and a staging environment that no longer validates the same chart previews use. The rule: everything stays in the declarative GitOps world unless network reality forces it out — only the production writer is forced out, because only a UoE host can reach the SMB share. The production-hosting specifics are rehearsed by the Wave-3 smoke test on the real slot instead. The agent image and code path are identical everywhere; only the mount and PdfBaseUrl differ, and the production-only pieces (CIFS mount, gatekeeper deploy) are covered by the Wave-3 smoke test.

Verified connectivity from arrnc-api (2026-08-11): SMB 445 to ecrf1.clinicaltrials.ed.ac.uk (= igmm-app2.igmm.ed.ac.uk) OK; S3 443 OK; RabbitMQ rabbitmq.camarades.net:5672 OK (5671 closed); cifs-utils installed.

6. Hosting on arrnc-api (production)¶

Managed via camaradesuk/server-config (Ansible) + camaradesuk/arrnc-api-deploy (gatekeeper). All patterns below have direct precedent in those repos.

6.1 Agent container¶

Gatekeeper-deployed daemon (precedent: mission-control-migration-worker-production, vars/projects.yml:748-765): nominal host_port: 8089 (nothing listens; 8070–8088 taken), no domains:, default service lifecycle, --restart unless-stopped, volume_path: /srv/data/syrf-pdf/production → /app/data, skip_web_acl: true, container_uid = the Dockerfile's dedicated non-root uid. Deploys run sudo /usr/local/bin/container-web-deploy syrf-pdf-agent-production ghcr.io/camaradesuk/syrf-pdf-agent:<tag> -e ... from a syrf deploy workflow.

Runner-boundary gate. CLAUDE.md currently rules that deploy/secret-bearing syrf jobs stay GitHub-hosted and prohibits SyRF-specific runner labels — written for the shared juniper CI pool. Hosting a deploy-only runner on arrnc-api (the established pattern every other tenant of that host uses: futurems-ecrf, edd-intake, deployment-portal) is a deliberate exception that must be reviewed and recorded before Wave 3: PR-3 updates the CLAUDE.md runner-boundary section to define the deploy-runner category (server-config-managed host, gatekeeper-restricted sudo, no build/test workloads, repo-scoped runner) as part of that review. Fallback if the review rejects it: a GitHub-hosted job invoking container-web-deploy on arrnc-api over SSH with a dedicated restricted key. See the Wave-3 prerequisite in the plan doc.

6.2 clamd companion¶

Ansible-managed container (mailpit precedent, roles/mailpit_staging/): new roles/clamav running clamav/clamav, joined to new bridge syrf-pdf-net (declared in docker_networks, non-internal so freshclam keeps outbound 443), no published ports — the agent reaches it as clamd:3310 via docker DNS. Signature DB on a host volume so restarts don't re-download. freshclam (bundled in the image) refreshes signatures on a timer and hot-reloads clamd. The role mounts a clamd.conf drop-in raising StreamMaxLength/MaxScanSize/MaxFileSize to cover the 500 MB entry cap (image defaults are far lower); the dev docker-compose clamd and the chart's clamd sidecar carry the same settings so limits behave identically everywhere.

6.3 CIFS mount¶

One new item in the existing cifs_mounts list (vars/projects.yml:972-994 precedent): //igmm-app2.igmm.ed.ac.uk/w3dev/csena/Camarades (the share root behind https://ecrf1.clinicaltrials.ed.ac.uk/camarades/; ecrf1 is a DNS alias of igmm-app2) → /srv/data/syrf-pdf/production/output, mount_user = agent uid, credentials from a new cifs/syrf-pdfs age scope (secrets_age_to_cifs bridge — zero new machinery, same pattern as the NeuroCARE share mounts). Credential handling: values stored in LastPass, then age-encrypted into vars/secrets.age.yaml in server-config — all configuration lives in server-config. The mountpoint nests inside the single gatekeeper bind volume; roles/docker_volumes ACLs are non-recursive (tasks/main.yml:22-80) so they don't touch the CIFS submount. Because a bind mount snapshots submounts at container start, a CIFS remount mid-flight is invisible to the running container — hence the sentinel file (§4.5) and a runbook note: restart the agent container after any manual remount.

6.4 Secrets and env¶

Non-secret env via vars/portal-env-overrides.yaml; the RabbitMQ password via the deploy workflow's -e args from GitHub environment secrets (the tenant-owned channel per arrnc-api-deploy README). CIFS credentials are host-level age secrets, never in the container env.

7. ADR-010 interaction (prerequisite)¶

Audited 2026-08-11: the production s3-notifier ACK cutover (cluster-gitops#344) merged but stalled — the ArgoCD app has been OutOfSync/Degraded since 2026-05-12 with the Function CR in ACK.Terminal "Resource already exists"; the production Lambda is still effectively Terraform's. Two primed hazards directly affect this feature:

The chart's Bucket manifest (never yet applied to production) declares a bucket-wide, unfiltered notification and no lifecycle rules. Its first successful production sync would drop Terraform's Projects/ prefix filter and the live Incomplete Multipart Cleanup lifecycle rule (ACK treats spec as whole desired state).
The production IAM role lacks the ignore_changes handover block staging has — a routine terraform apply and ACK fight over a live role.

This rollout therefore bundles a minimal unblock as Wave 0 (see plan doc): template notification filters + lifecycle rules in the chart (carrying the existing multipart rule), add the missing ignore_changes block, resolve the Function CR adoption conflict, and verify a no-op production sync. Our C4 lifecycle rule and the notifier code change then ride the normal promotion path. Full audit is recorded in the session memory and summarised in the plan doc's Wave 0.

8. Security model¶

No cloud credentials on UoE hosts (C3): presigned GET only, 12 h expiry.
No browser-held completion capability: presigned part URLs bind part number, exact Content-Length, and SHA-256, and accept a request start for 5 minutes; only the backend completes the object after verifying the stored manifest and whole-object metadata.
Bounded abandoned-upload lifetime: server-owned leases, monotonic states, and repeated verified cleanup make continuation impossible before capacity is released. The S3 incomplete-MPU lifecycle rule is defence in depth if application cleanup is unavailable.
No inbound connections to UoE: agent pulls S3 and AMQP outbound only.
All-or-nothing infection policy: an infected archive delivers nothing.
Server-side validation is authoritative: client checks are UX; the agent re-validates structure, scans every file, and checks magic bytes regardless of client behaviour.
Extraction hardening: Zip-Slip guards, entry/size caps, atomic writes.
AuthZ: dedicated BulkPdfUpload policy (A7); internal reporting flows via RabbitMQ only — no internal HTTP endpoints, no API keys (unlike #2373).
Known accepted risk (existing posture): AMQP to rabbitmq.camarades.net:5672 is plaintext over the public internet — same as the production Lambda today. A separate hardening PR adds AMQPS 5671; the agent flips its URI when it lands (D5).

9. Out of scope (MVP)¶

True resumability of a terminal/abandoned upload, adoption of parts into a retry, retry of individual failed files during agent processing, strict mode, migration of the existing 700 GB into the new layout, AMQPS enablement (separate PR), ADR-010 Phases 3-cleanup/6 beyond the minimal Wave-0 unblock.

10. Timing validation gate¶

The 20-minute lease, 60-second heartbeat, completed-part renewal boundary, 5-minute part-URL request-start validity, 600-second client request timeout, derived latest-signed-URL plus request duration cleanup boundary, 180-second completion-operation claim, 120-second server-owned storage timeout, and 30-second cleanup observation interval are configurable accepted defaults, not values to tune silently in production. Before the flag is enabled, staging validation must exercise a slow 1 GB upload and prove exact 16 MiB parts safely complete within 600 seconds, plus transient loss and bounded automatic retry, expected heartbeat jitter, and device sleep/background throttling. The acceptance criterion is that a transfer which can genuinely continue retains its session, while a severed non-resumable transfer converges through Aborting; after the derived quiescence boundary, two joint MPU-empty/object-absent observations at least one interval apart release capacity. If evidence shows the defaults unsafe, stop at that narrow decision and present revised values with evidence before changing them.

11. Open inputs¶

~~ecrf1 SMB share name/path~~ Resolved 2026-08-11: share root is \\igmm-app2.igmm.ed.ac.uk\w3dev\csena\Camarades. Credentials to be added to LastPass and age-encrypted into server-config (cifs/syrf-pdfs scope) before the Wave-3 production apply — NeuroCARE CIFS precedent.
Confirmed 2026-08-11: the IIS site serves new subdirectories without extra configuration.

12. Reference index¶

Discussions #2093/#2094/#2095/#2241/#1930 · epic #2223 · issues #2313, #2406/#2407, #2516–#2546,

2588/#2589 · prior PRs #2298 (closed), #2373 (to be superseded) · ADR-010 and its audit ·¶

docs/architecture/systematic-search-upload-flow.md (notifier reference; stale sections updated in Wave 2).