Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

The TimeText Corpus (TTC) is a multimodal dataset for forecasting, combining time series and text data, addressing the lack of well-curated datasets for this purpose. The Hybrid Multi-Modal Forecaster (Hybrid-MMF) model integrates numerical and textual data, showing competitive performance but marginal improvements over baselines in some cases. The dataset and code are available for further research. The text discusses using Large Language Models (LLMs) for time series forecasting, focusing on methods like LLMTime, Time-LLM, and Zhou et al.'s approach. Techniques include quantization and alignment for inputting numerical data into LLMs. The work introduces a multimodal forecasting model that combines textual and numerical time-series data, differing from Liu et al.'s Time-MMD in its ability to learn textual embeddings and its focus on multi-modal inputs for textual predictions. The dataset includes weather service discussions and recordings, and a medical dataset from MIMIC-III. The text outlines a hybrid model combining time-series and text data, using a multimodal forecasting framework that integrates time series and text data for enhanced forecasting accuracy. Experiments compare this model against text-only and numerical-only forecasting baselines, evaluating performance in both textual and numerical metrics.