Last week, our lead software engineer, Nelson Masuki and I presented at the MSRA Annual Conference to a room full of brilliant researchers, data scientists, and development practitioners from across Kenya and Africa. We were there to address a quietly growing dilemma in our field: the rise of synthetic data and its implications for the future of research, particularly in the regions we serve.
Our presentation was anchored in findings from our whitepaper that compared results from a traditional CATI survey data with synthetic outputs generated using several large language models (LLMs). The session was a mix of curiosity, concern, and critical thinking, especially when we demonstrated how off-the-mark synthetic data can be in places where cultural context, language, or ground realities are complex and rapidly changing.
We started the presentation by asking everyone to prompt their favourite AI app with some exact questions to model survey results. No two people in the hall got the same answers. Even though the prompt was exactly the same, and many people used the same apps on the same models, issue one.
The experiment
We then presented the findings from our experiments. Starting with a CATI survey of over 1,000 respondents in Kenya, we conducted a 25-minute study on several areas: food consumption, media and technology use, knowledge and attitudes toward AI, and views on humanitarian assistance. We then took the respondents’ demographic information (age, gender, rural-urban setting, education level, and ADM1 location) and created synthetic data respondents (SDRs) that exactly matched those respondents, and administered the same questionnaire across several LLMs and models (even did repeat cycles with newer, more advanced models). The differences were as varied as they were skewed – almost always wrong. Synthetic data failed the one true test of accuracy – the authentic voice of the people.
Many in the room had faced the same tension: global funding cuts, increasing demands for speed, and now, the allure of AI-generated insights that promise “just as good” without ever leaving a desk. But for those of us grounded in the realities of Africa, Asia, and Latin America, the idea of simulating the truth, of replacing real people with probabilistic patterns, doesn’t sit right.
This conversation, and others we had throughout the conference, affirmed a growing truth – AI will undoubtedly shape the future of research, but it must not replace real human input. At least not yet, and not in the parts of the world where truth on the ground doesn’t live in neatly labeled datasets. We cannot model what we’ve never measured.
Why Synthetic Data Can’t Replace Reality – Yet
Synthetic data is exactly what it sounds like: data that hasn’t been collected from real people, but generated algorithmically based on what models think the answers should be. In the research world, this typically involves creating simulated survey responses based on patterns identified from historical data, statistical models, or large language models (LLMs). While synthetic data can serve as a functional testing tool, and we are continually testing its utility in controlled experiments, it still falls short in several critical areas: it lacks ground truth, it missed nuance and context, and therefore it’s hard to trust.
And that’s precisely the problem.
In our side-by-side comparison of real survey responses and synthetic responses generated via LLMs, the differences were not subtle – they were foundational. The models guessed wrong on major indicators like unemployment levels, digital platform usage, and even simple household demographics.
I don’t believe this is just a statistical issue. It’s a context issue. In regions such as Africa, Asia, and Latin America, ground realities change rapidly. Behaviors, opinions, and access to services are highly local and deeply tied to culture, infrastructure, and lived experience. These are not things a language model trained predominantly on Western internet content can intuit.
Synthetic data can, indeed, be used
Synthetic data isn’t inherently bad. Lest you think we are anti-tech (which we can never be accused of), at GeoPoll, we do use synthetic data, just not as a replacement of real research. We use it to test survey logic and optimize scripts before fieldwork, simulate potential outcomes and spot logical contradictions in surveys, and experiment with framing by running parallel simulations before data collection.
And yes, we could generate synthetic datasets from scratch. With more than 50 million completed surveys across emerging markets, our dataset is arguably one of the most representative foundations for localized modeling.
However, we’ve also tested its limits, and the findings are clear: synthetic data cannot replace real, human-sourced insights in low-data environments. We don’t believe it’s ethical or accurate to replace fieldwork with simulations, especially when decisions about policy, investment, or aid are at stake. Synthetic data has its place. But in our view, it is not, and should not be, a shortcut for understanding real people in underrepresented regions. It’s a tool to augment research, not a replacement for it.
Data Equity Starts with Inclusion – GeoPoll AI Data Streams
There’s a significant reason this matters. While some are racing to build the next large language model (LLM), few are asking: What data are these models trained on? And who gets represented in those datasets?
GeoPoll is in this space, too. We now work with tech companies and research institutions to provide high-quality, consented data from underrepresented languages and regions, data used to train and fine-tune LLMs. GeoPoll AI Data Streams is designed to fill the gaps where global datasets fall short – to help build more inclusive, representative, and accurate LLMs that understand the contexts they seek to serve.
Because if AI is going to be truly global, it needs to learn from the entire globe, not just guess. We must ensure that the voices of real people, especially in emerging markets, shape both decisions and the technologies of tomorrow.
Contact us to learn more about GeoPoll AI Data Streams and how we use AI to power research.
