DIAGen: Diverse Image Augmentation with Generative Models

DIAGen is a novel image augmentation technique that enhances the generalization power of computer vision models by diversifying generations and improving semantic diversity. It builds upon DA-Fusion, incorporating Gaussian noise into object embeddings learned with Textual Inversion and using a pre-trained diffusion model for image generation. DIAGen also exploits a text-to-text generative model to guide image generation with varied class-specific prompts, and a weighting mechanism mitigates the impact of poorly generated samples. Across various datasets, DIAGen improves semantic diversity and classifier performance, especially with out-of-distribution samples, outperforming standard augmentations and the DA-Fusion baseline. The text discusses the challenges of limited data in few-shot learning scenarios, where data augmentation techniques are crucial for improving model generalization and robustness. DIAGen introduces a novel image augmentation technique using generative models, focusing on enhancing semantic diversity while maintaining high-quality images. It leverages the world-knowledge of a text-to-text generative model to obtain meaningful prompts, aiming to increase semantic diversity while addressing the fidelity-diversity trade-off. DIAGen outperforms DA-Fusion and standard augmentations in downstream classifier accuracy across multiple datasets in few-shot settings. DIAGen is an image augmentation technique that leverages generative models to enhance diversity in datasets. It has been tested on four datasets, including FOCUS, MS COCO, a custom COCO dataset, and an uncommon settings test set. DIAGen consistently outperforms standard augmentation methods and DA-Fusion, with accuracy gains of up to 10.5% points across the four datasets. In few-shot learning scenarios, DIAGen's ability to introduce additional semantic diversity is particularly beneficial, strengthening the model's generalization ability. The study by Lingenberg et al. focuses on the DIAGen pipeline, which comprises three components: embedding noise, LLM prompts, and a weighting mechanism. An ablation study was conducted to evaluate the individual impact of each component on classification accuracy. The results show that embedding noise significantly improves accuracy when only 2 examples per class are used for training. However, its positive effect diminishes with more examples per class. Combining embedding noise with the LLM prompts module leads to substantial benefits. The addition of the weighting mechanism further enhances accuracy, as it mitigates the trade-off between diversity and class fidelity, achieved by the embedding noise and LLM prompts. The full DIAGen method, incorporating all three components, demonstrates the greatest improvement in accuracy compared to the DA-Fusion baseline. DIAGen is an image augmentation technique that uses generative models to improve machine learning model performance. It combines noise and language model (LLM) prompts to create diverse, high-quality images. DIAGen outperforms using LLM prompts alone, especially when there are fewer examples per class. The method's effectiveness is not solely due to hyperparameter adjustments, as shown in an experiment comparing DIAGen to DA-Fusion with identical hyperparameters. DIAGen's true strength lies in the combination of its components, which can be fine-tuned for specific tasks to achieve better results. DIAGen also produces a higher level of semantic diversity in synthetic datasets compared to DA-Fusion, as visually observed. The text compares the DA-Fusion baseline and the DIAGen model in generating synthetic images. DIAGen produces more diverse images, as seen in the qualitative comparison, with noticeable differences in fine textural details. The table presents averaged precision and recall metrics between real and synthetic images generated by both models. DIAGen outperforms DA-Fusion in these metrics, especially when trained on smaller datasets. The results suggest that DIAGen enhances diversity effectively, as measured by precision and recall, even with limited training data. DIAGen is an image augmentation technique that enhances semantic diversity in datasets with limited labeled examples. It builds upon the DA-Fusion framework, incorporating three key components: introducing noise to class representations in the embedding space, enriching text prompts with semantically meaningful content, and using a weighting mechanism to reduce the influence of suboptimal generated images. This approach improves classification accuracy across various datasets, increases recall, and enables models to generalize to uncommon scenarios and edge cases, making it valuable for few-shot learning settings. The technique has been tested on three datasets, showing significant recall improvements and maintaining precision on the FOCUS dataset. However, precision drops on the Custom COCO dataset due to its small size and non-representative distribution. DIAGen's effectiveness is supported by its ability to balance fidelity and diversity in synthesized images. This project is partially funded by the European Research Council and the State of Hesse, Germany.