Reading music as a system

Melodious begins with a familiar action: photograph a printed score. What happens next is not ordinary optical character recognition. The platform must recover a structured musical document from spatial evidence, preserve enough detail for inspection, and return formats that musicians and notation software can actually use.

A notehead has meaning only through context. Its stem affects duration. A beam binds it to a rhythmic group. An accidental changes pitch. A clef establishes how its position on a staff should be interpreted. The central engineering challenge was therefore to move from isolated detections to a coherent score.

From image to notation

1. A React client accepts an uploaded score or sample and creates a transcription job.
2. FastAPI validates the request and schedules it through a bounded background worker.
3. A full-page YOLOv8 pass locates noteheads and contextual notation.
4. A tiled pass recovers thin symbols such as stems, beams, flags, and explicit accidentals.
5. The pipeline constructs plausible candidate edges and, when its checkpoint is available, applies the legacy MUSCIMA graph model.
6. Music logic derives notes, rests, pitch, partial rhythm, ties, slurs, and system structure.
7. Export services produce MusicXML, MIDI, event and detector JSON, relationship data, annotated overlays, engraved notation, and a ZIP bundle.
8. The client polls the job API, reports progress, and opens the completed artifacts in a review workspace.

The working product currently persists artifacts locally and keeps its job registry in memory. The API is Dockerized, with an AWS runbook and templates for ECR/ECS plus an S3/CloudFront frontend; those materials make the deployment path reproducible without implying that the public AWS deployment was completed from the audited workspace.

Decisions that changed the result

Early template-matching approaches were too brittle for variation in printing, photography, and page condition, so the detection layer moved to learned object detection. Thin symbols needed a second spatial strategy, which led to a tiled pass alongside the full-page context model. Relationship inference was then evaluated against deterministic local geometry using identical pages and candidate edges, isolating the value of the graph model rather than comparing mismatched pipelines.

The export layer is treated as a product boundary, not an afterthought. XML-derived holdout fixtures validate MusicXML and MIDI generation; intermediate detections and graph outputs remain inspectable; and the workspace surfaces model provenance, confidence, and warnings instead of hiding uncertainty.

What the user receives

• Original-versus-overlay comparison for visual inspection
• Editable MusicXML and in-browser engraved notation
• Playable MIDI with instrument and tempo controls
• Notes and ordered events for structured review
• Detector payloads, relationship data, provenance, confidence, and warnings
• A complete downloadable ZIP bundle

Outcome

On the frozen held-out detector test, Melodious reached 0.852 mAP@0.5, beating published DeepScoresV2 benchmarks including Faster R-CNN + HRNet (0.799) and Deep Watershed Detector (0.503). It also reached 0.707 mAP@0.5:0.95 and 0.839 F1@0.5. Its head spans 136 taxonomy classes; the local annotations cover 115 classes in total, with 110 supported in the final test.

On graph validation, the relationship model reached 0.759 positive-class macro F1, roughly a 22% relative uplift over deterministic local geometry on the same 14 pages and 48,174 candidate edges. The export boundary produced valid MusicXML, successful MIDI, and a successful page result for 14 of 14 XML-derived holdout fixtures.

The result is an inspectable product path from an uploaded page to detections, musical structure, playback, and reusable score artifacts. Measure repair, voice separation, exact rhythm, and barline-constrained normalization remain active engineering limits rather than hidden claims.