TSI-Bench: Benchmarking Time Series Imputation

Wenjie Du, Jun Wang, Linglong Qian, Yiyuan Yang, Fanxing Liu, Zepu Wang, Zina Ibrahim, Haoxin Liu, Zhiyuan Zhao, Yingjie Zhou, Wenjia Wang, Kaize Ding, Yuxuan Liang, B. Aditya Prakash, Qingsong Wen·June 18, 2024

Summary

TSI-Bench is a comprehensive benchmark platform for time series imputation using deep learning, designed to standardize evaluation and address the lack of a unified framework. It evaluates 28 algorithms across seven categories on eight diverse datasets from various domains, focusing on missingness ratios, patterns, and forecasting adaptations. Experiments show that different patterns and rates significantly impact model performance, with forecasting-based models like Informer and BRITS demonstrating strong results. TSI-Bench also highlights the importance of data preprocessing, missingness simulation, and the influence of domain-specific data processing on downstream task performance. The platform, available on GitHub, encourages researchers to tailor methods to specific problems and datasets, providing valuable insights for future time series imputation research.

Key findings

13

Paper digest

What problem does the paper attempt to solve? Is this a new problem?

The paper "TSI-Bench: Benchmarking Time Series Imputation" aims to address the challenge of time series imputation, specifically focusing on evaluating different imputation methods under various missing data scenarios . This problem is not entirely new, as imputation methods have been developed and used in the past to handle missing values in time series data. However, the paper contributes by providing a benchmarking framework to systematically compare the performance of multiple imputation techniques under different missing data patterns, such as point missing, subsequence missing, and block missing . The paper's approach helps in assessing the effectiveness of existing imputation methods and potentially identifying new strategies to improve imputation accuracy in time series datasets.


What scientific hypothesis does this paper seek to validate?

This paper aims to validate the performance of various imputation methods for time series data with missing values . The study evaluates different imputation techniques under scenarios of 10%, 50%, and 90% missing data, including point missing, subsequence missing, and block missing . The research investigates how well these methods can estimate missing values in time series datasets, which is crucial for improving data quality and downstream tasks such as classification and regression . The paper provides a comprehensive analysis of the imputation results and their impact on the overall performance of time series data analysis, highlighting the importance of accurate imputation methods in handling missing data effectively .


What new ideas, methods, or models does the paper propose? What are the characteristics and advantages compared to previous methods?

The paper "TSI-Bench: Benchmarking Time Series Imputation" introduces several new ideas, methods, and models for time series imputation .

  1. Imputation Scenarios:

    • The paper explores different missing data scenarios, including 10% point missing, 50% point missing, 90% point missing, 50% subsequence missing, and 50% block missing .
  2. Imputation Performance:

    • It evaluates the imputation performance under these scenarios and discusses the challenges faced in estimating missing values in extreme scenarios .
  3. Imputation Methods:

    • The paper presents various imputation methods such as SAITS, ETSformer, PatchTST, Transformer, BRITS, MRNN, GRUD, FiLM, CSDI, US-GAN, GP-VAE, SCINet, StemGNN, FreTS, and Koopa .
  4. Comparison and Analysis:

    • It compares the performance of these methods in terms of imputation error and their ability to provide reasonable values for missing components, ultimately improving data quality .
  5. Visualization:

    • The paper includes visualizations of imputation examples by different methods, showcasing the effectiveness of these methods in handling missing data in time series . The paper "TSI-Bench: Benchmarking Time Series Imputation" introduces several novel methods and models for time series imputation, each with unique characteristics and advantages compared to previous methods .
  6. Crossformer:

    • Characteristics: Crossformer utilizes cross-dimensional attention to model intricate dependencies within multivariate time series data, achieving good performance in complex forecasting scenarios .
  7. Informer:

    • Advantages: Informer enhances efficiency in long time series forecasting by employing a self-attention distillation mechanism, reducing redundant information while maintaining forecasting accuracy .
  8. Autoformer:

    • Characteristics: Autoformer introduces a novel decomposition architecture with an auto-correlation mechanism, effectively capturing both seasonal and trend patterns in time series forecasting tasks .
  9. Pyraformer:

    • Advantages: Pyraformer is designed for long-term time series forecasting, utilizing a pyramid attention structure that efficiently captures temporal dependencies at multiple scales .
  10. Transformer:

    • Characteristics: Transformer introduces the self-attention mechanism, enabling the processing of sequential data by attending to different positions within the sequence simultaneously, leading to significant advancements in natural language processing and time series fields .
  11. BRITS:

    • Advantages: BRITS employs a bidirectional recurrent imputation strategy to handle missing values in time series, improving forecasting accuracy through iterative refinement .
  12. CSDI:

    • Characteristics: CSDI leverages conditional score-based diffusion models for accurate imputation and generation of missing values in time series applications .
  13. US-GAN:

    • Advantages: US-GAN integrates a classifier in a semi-supervised generative adversarial network to enhance imputation of missing values in multivariate time series, leveraging observed data and label information .
  14. GP-VAE:

    • Characteristics: GP-VAE proposes a deep sequential latent variable model combining VAE with a structured variational approximation to achieve non-linear dimensionality reduction and imputation, providing interpretable uncertainty estimates and improved imputation smoothness .

Do any related researches exist? Who are the noteworthy researchers on this topic in this field?What is the key to the solution mentioned in the paper?

Several related research studies exist in the field of time series imputation. Noteworthy researchers in this area include those who have contributed to the TSI-Bench paper, such as the authors who conducted benchmarking on time series imputation methods . The key solution mentioned in the paper involves evaluating the performance of imputation algorithms using different missing patterns, including point, subsequence, and block missing, and assessing their effectiveness based on metrics like MAE, MSE, and MRE .


How were the experiments in the paper designed?

The experiments in the paper were designed to evaluate time series imputation algorithms across different settings using a comprehensive benchmark suite called TSI-Bench . The experiments involved 28 algorithms and 8 datasets with diverse missingness scenarios . The TSI-Bench pipeline standardized experimental settings to enable fair evaluation of imputation algorithms and identify insights into the influence of missingness ratios and patterns on model performance . Additionally, the experiments explored the imputation effects of various algorithms under 5 missing patterns and evaluated the performance of downstream tasks after imputation . The study conducted a total of 34,804 experiments to assess the effectiveness of TSI-Bench in diverse downstream tasks and its potential to advance time series imputation research and analysis .


What is the dataset used for quantitative evaluation? Is the code open source?

The dataset used for quantitative evaluation in the study is the ETT_h1 dataset . The code for the evaluation methods is not explicitly mentioned as open source in the provided context. If you are interested in accessing the code, it would be advisable to refer to the original source or contact the authors of the study for more information regarding the availability of the code .


Do the experiments and results in the paper provide good support for the scientific hypotheses that need to be verified? Please analyze.

The experiments and results presented in the paper provide substantial support for the scientific hypotheses that need to be verified. The paper conducts experiments on time series imputation using various methods and evaluates their performance based on different missing data scenarios . The results showcase the effectiveness of different imputation methods in handling missing values in time series data, demonstrating their ability to provide reasonable values for the missing components and improve data quality . The experiments cover scenarios such as 10% point missing, 50% point missing, 90% point missing, 50% subsequence missing, and 50% block missing, which are crucial in assessing the robustness and efficacy of the imputation methods .

Moreover, the paper includes detailed tables and figures that illustrate the performance of each imputation method under different missing data settings . The results show the imputation error rates, inference times, and the ability of the methods to estimate missing values accurately in challenging scenarios like 90% point missing and 50% block missing . Overall, the comprehensive experimental setup and the detailed analysis of results provide strong empirical evidence to support and validate the scientific hypotheses related to time series imputation methods .


What are the contributions of this paper?

The paper "TSI-Bench: Benchmarking Time Series Imputation" makes several contributions in the field of time series imputation :

  • It provides a benchmarking framework for evaluating different time series imputation methods.
  • The paper introduces and compares various imputation methods such as Mean, Median, LOCF (Last Observation Carried Forward), Linear interpolation, Crossformer, Informer, Autoformer, Pyraformer, LOCF, and Linear .
  • It evaluates the performance of these methods using metrics like MAE (Mean Absolute Error), MSE (Mean Square Error), and MRE (Mean Relative Error) .
  • The study explores different missing patterns in time series data, including point missing, subsequence missing, and block missing, to assess the effectiveness of imputation algorithms under various scenarios .
  • The paper visualizes the imputation results of different methods through heatmaps and provides insights into the challenges and performance of imputation techniques under different missing data patterns .
  • It presents experimental results under different missing data settings, such as 10% point missing, 50% point missing, 90% point missing, 50% subsequence missing, and 50% block missing, to analyze the imputation performance in varying degrees of data incompleteness .

What work can be continued in depth?

To delve deeper into the benchmarking of time series imputation, further exploration can focus on the following aspects :

  1. Detailed Model Analysis: Conduct a comprehensive analysis of the various time series imputation models benchmarked in the study, such as Transformer-based models like iTransformer, SAITS, Nonstationary, ETSformer, PatchTST, and Crossformer, to understand their specific architectures, strengths, and weaknesses in handling missing data and forecasting accuracy.

  2. Performance Evaluation: Further investigate the performance metrics of the models across different datasets and missing patterns to assess their effectiveness in imputing missing values and their impact on downstream tasks like forecasting. Analyze metrics like PR_AUC and ROC_AUC weighted scores for models like iTransformer, SAITS, Nonstationary, ETSformer, PatchTST, Crossformer, and Informer to gain insights into their predictive capabilities.

  3. Adaptation Paradigm: Explore the adaptation paradigm proposed in the study, where forecasting models are transformed for imputation tasks. Investigate how the imputation framework from SAITS is utilized to tailor forecasting models for imputing missing values in time series data, focusing on the input-output processing modifications and joint-optimization training approach employed.

  4. Development Environment: Delve into the details of the development environment used for the experiments, including the hardware specifications, software environment (Ubuntu 12.3.0), and the HPC platform at King’s College London CREATE. Understanding the computational resources and tools utilized can provide insights into the reproducibility and scalability of the study.

  5. Datasets Preprocessing: Further explore the preprocessing details of datasets like BeijingAir, ItalyAir, PeMS, Electricity, and ETT_h1 to understand how the data was split into training, validation, and test sets based on time periods. Investigate the sliding window function applied to generate data samples and ensure no data leakage in the experiments.

By delving deeper into these aspects, researchers can gain a more comprehensive understanding of the benchmarked time series imputation models, their performance on different datasets, the adaptation paradigm used, the experimental setup, and the preprocessing techniques applied to the data.

Tables

7

Introduction
Background
Evolution of time series imputation methods
Need for a unified evaluation platform
Objective
Standardize evaluation process
Address research gaps in unified framework
Methodology
Data Collection
Datasets
Description of eight diverse datasets (domains: finance, healthcare, climate, etc.)
Missingness characteristics (patterns and ratios)
Data preprocessing techniques applied
Algorithms and Categories
Overview of 28 algorithms
Seven categories: basic imputation, forecasting-based, deep learning, etc.
Selection criteria for inclusion
Performance Metrics
Evaluation metrics (accuracy, RMSE, etc.)
Importance of forecasting accuracy and adaptability
Experiments and Findings
Impact of missingness patterns and rates on model performance
Informer and BRITS: standout models
Data preprocessing techniques' influence on performance
Missingness Simulation
Techniques used for simulating missing values
Effect on model performance
Domain-Specific Considerations
Importance of domain knowledge in data processing
Case studies on tailored methods
Platform and Usage
GitHub Repository
Access and structure of the TSI-Bench platform
Code examples and documentation
Encouraging Research
Recommendations for researchers to adapt methods
Value for future time series imputation research
Conclusion
Summary of key insights
Future directions for TSI-Bench and time series imputation research
Basic info
papers
machine learning
artificial intelligence
Advanced features
Insights
What is TSI-Bench primarily designed for?
Which types of models tend to perform well in TSI-Bench's experiments?
How many algorithms and categories does TSI-Bench evaluate?
What kind of information does TSI-Bench emphasize in understanding time series imputation performance?

TSI-Bench: Benchmarking Time Series Imputation

Wenjie Du, Jun Wang, Linglong Qian, Yiyuan Yang, Fanxing Liu, Zepu Wang, Zina Ibrahim, Haoxin Liu, Zhiyuan Zhao, Yingjie Zhou, Wenjia Wang, Kaize Ding, Yuxuan Liang, B. Aditya Prakash, Qingsong Wen·June 18, 2024

Summary

TSI-Bench is a comprehensive benchmark platform for time series imputation using deep learning, designed to standardize evaluation and address the lack of a unified framework. It evaluates 28 algorithms across seven categories on eight diverse datasets from various domains, focusing on missingness ratios, patterns, and forecasting adaptations. Experiments show that different patterns and rates significantly impact model performance, with forecasting-based models like Informer and BRITS demonstrating strong results. TSI-Bench also highlights the importance of data preprocessing, missingness simulation, and the influence of domain-specific data processing on downstream task performance. The platform, available on GitHub, encourages researchers to tailor methods to specific problems and datasets, providing valuable insights for future time series imputation research.
Mind map
Data preprocessing techniques applied
Missingness characteristics (patterns and ratios)
Description of eight diverse datasets (domains: finance, healthcare, climate, etc.)
Value for future time series imputation research
Recommendations for researchers to adapt methods
Code examples and documentation
Access and structure of the TSI-Bench platform
Case studies on tailored methods
Importance of domain knowledge in data processing
Effect on model performance
Techniques used for simulating missing values
Data preprocessing techniques' influence on performance
Informer and BRITS: standout models
Impact of missingness patterns and rates on model performance
Importance of forecasting accuracy and adaptability
Evaluation metrics (accuracy, RMSE, etc.)
Selection criteria for inclusion
Seven categories: basic imputation, forecasting-based, deep learning, etc.
Overview of 28 algorithms
Datasets
Address research gaps in unified framework
Standardize evaluation process
Need for a unified evaluation platform
Evolution of time series imputation methods
Future directions for TSI-Bench and time series imputation research
Summary of key insights
Encouraging Research
GitHub Repository
Domain-Specific Considerations
Missingness Simulation
Experiments and Findings
Performance Metrics
Algorithms and Categories
Data Collection
Objective
Background
Conclusion
Platform and Usage
Methodology
Introduction
Outline
Introduction
Background
Evolution of time series imputation methods
Need for a unified evaluation platform
Objective
Standardize evaluation process
Address research gaps in unified framework
Methodology
Data Collection
Datasets
Description of eight diverse datasets (domains: finance, healthcare, climate, etc.)
Missingness characteristics (patterns and ratios)
Data preprocessing techniques applied
Algorithms and Categories
Overview of 28 algorithms
Seven categories: basic imputation, forecasting-based, deep learning, etc.
Selection criteria for inclusion
Performance Metrics
Evaluation metrics (accuracy, RMSE, etc.)
Importance of forecasting accuracy and adaptability
Experiments and Findings
Impact of missingness patterns and rates on model performance
Informer and BRITS: standout models
Data preprocessing techniques' influence on performance
Missingness Simulation
Techniques used for simulating missing values
Effect on model performance
Domain-Specific Considerations
Importance of domain knowledge in data processing
Case studies on tailored methods
Platform and Usage
GitHub Repository
Access and structure of the TSI-Bench platform
Code examples and documentation
Encouraging Research
Recommendations for researchers to adapt methods
Value for future time series imputation research
Conclusion
Summary of key insights
Future directions for TSI-Bench and time series imputation research
Key findings
13

Paper digest

What problem does the paper attempt to solve? Is this a new problem?

The paper "TSI-Bench: Benchmarking Time Series Imputation" aims to address the challenge of time series imputation, specifically focusing on evaluating different imputation methods under various missing data scenarios . This problem is not entirely new, as imputation methods have been developed and used in the past to handle missing values in time series data. However, the paper contributes by providing a benchmarking framework to systematically compare the performance of multiple imputation techniques under different missing data patterns, such as point missing, subsequence missing, and block missing . The paper's approach helps in assessing the effectiveness of existing imputation methods and potentially identifying new strategies to improve imputation accuracy in time series datasets.


What scientific hypothesis does this paper seek to validate?

This paper aims to validate the performance of various imputation methods for time series data with missing values . The study evaluates different imputation techniques under scenarios of 10%, 50%, and 90% missing data, including point missing, subsequence missing, and block missing . The research investigates how well these methods can estimate missing values in time series datasets, which is crucial for improving data quality and downstream tasks such as classification and regression . The paper provides a comprehensive analysis of the imputation results and their impact on the overall performance of time series data analysis, highlighting the importance of accurate imputation methods in handling missing data effectively .


What new ideas, methods, or models does the paper propose? What are the characteristics and advantages compared to previous methods?

The paper "TSI-Bench: Benchmarking Time Series Imputation" introduces several new ideas, methods, and models for time series imputation .

  1. Imputation Scenarios:

    • The paper explores different missing data scenarios, including 10% point missing, 50% point missing, 90% point missing, 50% subsequence missing, and 50% block missing .
  2. Imputation Performance:

    • It evaluates the imputation performance under these scenarios and discusses the challenges faced in estimating missing values in extreme scenarios .
  3. Imputation Methods:

    • The paper presents various imputation methods such as SAITS, ETSformer, PatchTST, Transformer, BRITS, MRNN, GRUD, FiLM, CSDI, US-GAN, GP-VAE, SCINet, StemGNN, FreTS, and Koopa .
  4. Comparison and Analysis:

    • It compares the performance of these methods in terms of imputation error and their ability to provide reasonable values for missing components, ultimately improving data quality .
  5. Visualization:

    • The paper includes visualizations of imputation examples by different methods, showcasing the effectiveness of these methods in handling missing data in time series . The paper "TSI-Bench: Benchmarking Time Series Imputation" introduces several novel methods and models for time series imputation, each with unique characteristics and advantages compared to previous methods .
  6. Crossformer:

    • Characteristics: Crossformer utilizes cross-dimensional attention to model intricate dependencies within multivariate time series data, achieving good performance in complex forecasting scenarios .
  7. Informer:

    • Advantages: Informer enhances efficiency in long time series forecasting by employing a self-attention distillation mechanism, reducing redundant information while maintaining forecasting accuracy .
  8. Autoformer:

    • Characteristics: Autoformer introduces a novel decomposition architecture with an auto-correlation mechanism, effectively capturing both seasonal and trend patterns in time series forecasting tasks .
  9. Pyraformer:

    • Advantages: Pyraformer is designed for long-term time series forecasting, utilizing a pyramid attention structure that efficiently captures temporal dependencies at multiple scales .
  10. Transformer:

    • Characteristics: Transformer introduces the self-attention mechanism, enabling the processing of sequential data by attending to different positions within the sequence simultaneously, leading to significant advancements in natural language processing and time series fields .
  11. BRITS:

    • Advantages: BRITS employs a bidirectional recurrent imputation strategy to handle missing values in time series, improving forecasting accuracy through iterative refinement .
  12. CSDI:

    • Characteristics: CSDI leverages conditional score-based diffusion models for accurate imputation and generation of missing values in time series applications .
  13. US-GAN:

    • Advantages: US-GAN integrates a classifier in a semi-supervised generative adversarial network to enhance imputation of missing values in multivariate time series, leveraging observed data and label information .
  14. GP-VAE:

    • Characteristics: GP-VAE proposes a deep sequential latent variable model combining VAE with a structured variational approximation to achieve non-linear dimensionality reduction and imputation, providing interpretable uncertainty estimates and improved imputation smoothness .

Do any related researches exist? Who are the noteworthy researchers on this topic in this field?What is the key to the solution mentioned in the paper?

Several related research studies exist in the field of time series imputation. Noteworthy researchers in this area include those who have contributed to the TSI-Bench paper, such as the authors who conducted benchmarking on time series imputation methods . The key solution mentioned in the paper involves evaluating the performance of imputation algorithms using different missing patterns, including point, subsequence, and block missing, and assessing their effectiveness based on metrics like MAE, MSE, and MRE .


How were the experiments in the paper designed?

The experiments in the paper were designed to evaluate time series imputation algorithms across different settings using a comprehensive benchmark suite called TSI-Bench . The experiments involved 28 algorithms and 8 datasets with diverse missingness scenarios . The TSI-Bench pipeline standardized experimental settings to enable fair evaluation of imputation algorithms and identify insights into the influence of missingness ratios and patterns on model performance . Additionally, the experiments explored the imputation effects of various algorithms under 5 missing patterns and evaluated the performance of downstream tasks after imputation . The study conducted a total of 34,804 experiments to assess the effectiveness of TSI-Bench in diverse downstream tasks and its potential to advance time series imputation research and analysis .


What is the dataset used for quantitative evaluation? Is the code open source?

The dataset used for quantitative evaluation in the study is the ETT_h1 dataset . The code for the evaluation methods is not explicitly mentioned as open source in the provided context. If you are interested in accessing the code, it would be advisable to refer to the original source or contact the authors of the study for more information regarding the availability of the code .


Do the experiments and results in the paper provide good support for the scientific hypotheses that need to be verified? Please analyze.

The experiments and results presented in the paper provide substantial support for the scientific hypotheses that need to be verified. The paper conducts experiments on time series imputation using various methods and evaluates their performance based on different missing data scenarios . The results showcase the effectiveness of different imputation methods in handling missing values in time series data, demonstrating their ability to provide reasonable values for the missing components and improve data quality . The experiments cover scenarios such as 10% point missing, 50% point missing, 90% point missing, 50% subsequence missing, and 50% block missing, which are crucial in assessing the robustness and efficacy of the imputation methods .

Moreover, the paper includes detailed tables and figures that illustrate the performance of each imputation method under different missing data settings . The results show the imputation error rates, inference times, and the ability of the methods to estimate missing values accurately in challenging scenarios like 90% point missing and 50% block missing . Overall, the comprehensive experimental setup and the detailed analysis of results provide strong empirical evidence to support and validate the scientific hypotheses related to time series imputation methods .


What are the contributions of this paper?

The paper "TSI-Bench: Benchmarking Time Series Imputation" makes several contributions in the field of time series imputation :

  • It provides a benchmarking framework for evaluating different time series imputation methods.
  • The paper introduces and compares various imputation methods such as Mean, Median, LOCF (Last Observation Carried Forward), Linear interpolation, Crossformer, Informer, Autoformer, Pyraformer, LOCF, and Linear .
  • It evaluates the performance of these methods using metrics like MAE (Mean Absolute Error), MSE (Mean Square Error), and MRE (Mean Relative Error) .
  • The study explores different missing patterns in time series data, including point missing, subsequence missing, and block missing, to assess the effectiveness of imputation algorithms under various scenarios .
  • The paper visualizes the imputation results of different methods through heatmaps and provides insights into the challenges and performance of imputation techniques under different missing data patterns .
  • It presents experimental results under different missing data settings, such as 10% point missing, 50% point missing, 90% point missing, 50% subsequence missing, and 50% block missing, to analyze the imputation performance in varying degrees of data incompleteness .

What work can be continued in depth?

To delve deeper into the benchmarking of time series imputation, further exploration can focus on the following aspects :

  1. Detailed Model Analysis: Conduct a comprehensive analysis of the various time series imputation models benchmarked in the study, such as Transformer-based models like iTransformer, SAITS, Nonstationary, ETSformer, PatchTST, and Crossformer, to understand their specific architectures, strengths, and weaknesses in handling missing data and forecasting accuracy.

  2. Performance Evaluation: Further investigate the performance metrics of the models across different datasets and missing patterns to assess their effectiveness in imputing missing values and their impact on downstream tasks like forecasting. Analyze metrics like PR_AUC and ROC_AUC weighted scores for models like iTransformer, SAITS, Nonstationary, ETSformer, PatchTST, Crossformer, and Informer to gain insights into their predictive capabilities.

  3. Adaptation Paradigm: Explore the adaptation paradigm proposed in the study, where forecasting models are transformed for imputation tasks. Investigate how the imputation framework from SAITS is utilized to tailor forecasting models for imputing missing values in time series data, focusing on the input-output processing modifications and joint-optimization training approach employed.

  4. Development Environment: Delve into the details of the development environment used for the experiments, including the hardware specifications, software environment (Ubuntu 12.3.0), and the HPC platform at King’s College London CREATE. Understanding the computational resources and tools utilized can provide insights into the reproducibility and scalability of the study.

  5. Datasets Preprocessing: Further explore the preprocessing details of datasets like BeijingAir, ItalyAir, PeMS, Electricity, and ETT_h1 to understand how the data was split into training, validation, and test sets based on time periods. Investigate the sliding window function applied to generate data samples and ensure no data leakage in the experiments.

By delving deeper into these aspects, researchers can gain a more comprehensive understanding of the benchmarked time series imputation models, their performance on different datasets, the adaptation paradigm used, the experimental setup, and the preprocessing techniques applied to the data.

Tables
7
Scan the QR code to ask more questions about the paper
© 2025 Powerdrill. All rights reserved.