chore(deployment,website,tests): remove dummy pipeline and organisms

[theosanderson]

Remove Dummy Organisms and Dummy Preprocessing Pipeline

Summary

This PR removes the dummy-organism, dummy-organism-with-files, and the entire preprocessing/dummy directory while retaining all test coverage by migrating to use real organisms with nextclade preprocessing.

Changes

Kubernetes/Helm Configuration

• Removed dummy-organism from values.yaml - tests now use ebola-sudan instead
• Renamed dummy-organism-with-files to test-organism-files - this organism is used for file upload testing and now uses nextclade preprocessing instead of the dummy preprocessing pipeline
• The test-organism-files organism:
  • Has consensusSequences: false (metadata-only, no sequences required)
  • Has file uploads enabled (raw_reads category)
  • Uses preprocessing-nextclade with no genes configured (for metadata-only processing)

Preprocessing Changes

Added file preservation to the nextclade preprocessing pipeline so that files attached to submissions are carried through to the processed output:

• datatypes.py: Added files field to UnprocessedData class
• backend.py: Extract files from backend JSON response when parsing unprocessed data
• prepro.py: Pass files through to ProcessedData in the no-alignment code path

Integration Tests

Updated tests to use real organisms instead of dummy organisms:

• file-sharing.spec.ts: Changed ORGANISM_URL_NAME from dummy-organism-with-files to test-organism-files
• column-visibility.dependent.spec.ts: Changed from "Test Dummy Organism" to "Ebola Sudan"
• landing-page.spec.ts: Changed featured organism to "Ebola Sudan"
• navigation.spec.ts: Changed organism to "Ebola Sudan"
• authentication.spec.ts: Changed URL from /dummy-organism/... to /ebola-sudan/...
• submission.page.ts: Renamed fillSubmissionFormDummyOrganism to fillSubmissionFormFilesOrganism

Website

Updated the dev example data generator in SequenceEntryUploadComponent.tsx:

• Now shows on ebola-sudan pages instead of dummy-organism pages
• Generates ebola-sudan compatible metadata (submissionId, sampleCollectionDate, geoLocCountry, authorAffiliations)
• Uses a valid ebola Sudan sequence for the sequence file

Deleted Files

• preprocessing/dummy/ (entire directory including main.py, Dockerfile, mock-sequences.json, etc.)
• .github/workflows/preprocessing-dummy-image.yml

CI/Workflow Updates

• Removed trigger-dummy-preprocessing job from .github/workflows/build-arm-images.yaml

🚀 Preview: https://dummy2.loculus.org

[chaoran-chen]

Very nice, thanks @theosanderson! Before reviewing the code in detail, I have a few general comments.

[chaoran-chen]

There're (at least?) two mentions of the dummy pipeline in the docs that should be removed

• https://loculus.org/for-administrators/build-new-preprocessing-pipeline/#_top
• https://loculus.org/for-administrators/setup-with-kubernetes/#organism-configuration

[theosanderson]

Thanks for the review! Will address these

[chaoran-chen]

It would be great to have a short migration guide just as part of the PR description about how to migrate from the dummy pipeline to the Nextclade one. As the dummy pipeline was the only one that supported files, I've used it whenever showing someone how to set up an instance with files, so it would be good to have a link to send to people for migrating.

[theosanderson]

The dummy pipeline is explicitly described as just for running tests and demonstrations, rather than for end-users, and I think that has been our consensus understanding of it as developers (hence tolerating various bugs), so I don't really consider this supported behaviour. At the time of implementation I did suggest implementing the functionality directly in the Nextclade pipeline. As below, I've no problem with us keeping the dummy pipeline around for weeks or a month, and of course I have no problem with anyone writing such a guide for when we remove it but atm I wouldn't personally plan to.

[chaoran-chen]

Similar to the point above, as we've so far been using the dummy pipeline whenever we needed files, including in our live wastewater instance, it would be great if we can first test that the Nextclade pipeline is working well before the dummy pipeline gets dropped. Could we first merge the new Nextclade pipeline feature and changes to the Loculus previews but leave the old pipeline code and image for two more weeks after merging this PR before removing?

[theosanderson]

Yes, there's no rush to remove the old version!

[theosanderson]

This is hardcoded into Loculus so would appear on PPX for ebola sudan! Must fix :)