What can LLM tell us about cities?

The study investigates large language models' (LLMs) capacity to provide global city knowledge, focusing on their predictive accuracy and ability to identify latent features. LLMs demonstrate varying degrees of information across cities, with improved accuracy when trained on LLM-derived features. They offer new opportunities for data-driven city research, potentially enabling traffic pattern modeling for any city. Challenges in collecting city data globally highlight LLMs' potential to overcome these issues by providing broad city understanding. The text discusses using LLMs for city tasks, aiming to demonstrate their ability to achieve comparable accuracy to traditional feature engineering with less effort. LLMs are utilized in three categories: as prediction models, fine-tuned for specific tasks, and to enhance feature extraction. The research introduces two methods: directly asking LLMs for target variable values and extracting features to train machine learning models. Experiments are conducted on open-source LLMs, with notable results using Llama3.1-8B-Instruct. The study explores LLMs' knowledge about cities, focusing on their ability to generate features that maintain relative order compared to ground truth data. GPT-4o outperforms other models in generating Methane data for Chinese cities, with errors ranging from 8% to 19% higher than GPT-4o. Model size impact was minimal in NYC 311 data analysis. Prompts in English showed better performance for global cities, while Chinese and English were comparable for Chinese cities. LLMs generally know information about various cities, but generate generic or random answers for less common knowledge. Explicit feature extraction is more effective and versatile for capturing information across models.