Cue Point Estimation using Object Detection

CUE-DETR is a novel method for automatic cue point estimation in DJ mixing, interpreted as a computer vision object detection task. It is based on a pre-trained object detection transformer fine-tuned on a new dataset containing 21,000 manually annotated cue points from human experts and metronome information for nearly 5,000 tracks, making it 35 times larger than the previous dataset. CUE-DETR demonstrates increased precision in retrieving cue point positions and high adherence to phrasing, a type of high-level music structure. The method outperforms previous approaches without requiring detailed rule sets based on low-level audio information. The paper also provides the code, model checkpoints, and dataset for further research. The paper discusses methods for identifying "cue points" in music, which are significant moments or sections that can be used for tasks like DJ mixing. It highlights the importance of understanding a track's structure, which can be enhanced by analyzing crowd-sourced data from streaming services. The text also mentions the use of learning-based algorithms to directly search for musical highlights, which can provide useful information about a track's structure. The accuracy of algorithmically chosen cue points depends on the rule set used, and adding more rules can introduce a trade-off between the number of correctly estimated cue points and their correctness. The text then introduces the Automix system, which uses a rule-based algorithm for cue point estimation and includes a validation dataset. It also discusses the limitations of using existing DJ mixes for cue point estimation and the use of machine learning techniques, such as convolutional neural networks (CNNs) and transformers, for structural analysis. The proposed CUE-DETR architecture processes Mel spectrograms to identify cue points, represented as bounding boxes, during training. Inference images move across the spectrogram using a sliding window, and the highest scoring positions are selected with a minimum confidence threshold of 0.9. A radius of 16 or 8 bars is enforced between predicted cue points to improve precision. The model is initialized with pre-trained weights from DETR, and training is conducted on 101 tracks not used in the training or validation splits. The evaluation compares the model's performance against "Mixed In Key 10" and Automix, using metrics such as hit rate, precision, recall, F1-score, and Average Precision (AP) scores. The impact of the bounding box width on prediction quality is also investigated. CUE-DETR, an object detection model fine-tuned on Mel spectrograms, excels in estimating cue points in EDM tracks, outperforming previous methods in precision, recall, and F1-score. It shows strong adherence to ground-truth, with better phrase alignment and fewer false positive predictions. CUE-DETR predicts cue points with the highest adherence to ground-truth, and its cosine similarity score for quantized predictions is the highest among the methods compared. The model's metronome-agnostic approach, using fixed distances relative to the dataset median tempo, yields higher precision compared to other methods. However, for more diverse music styles, incorporating tempo and beat grid information could improve performance. Future work aims to investigate the method's performance across a broader domain of electronic music and with annotations from different types of DJs. The paper references various studies and publications on advancements in automatic DJ systems and music information retrieval, focusing on aspects such as rhythm, meter, and musical design in electronic dance music. Key areas of research include the development of manually annotated datasets for cue points in music, automatic sequencing and seamless mixing of dance music tracks, full-automatic DJ mixing systems with optimal tempo adjustments, creation of an automated DJ system for drum and bass music, automatic detection of cue points for DJ mixing, and automatic audio segmentation using a measure of audio novelty.