The question
From a model result to a defensible claim
Researchers can agree on the data and act in good faith, yet reach different conclusions about whether an information source improves prediction. The choices between those two points deserve close attention.
This paper examines how transformations, benchmarks and evaluation rules shape predictive findings. Its connection to model risk lies in the interpretation: the assumptions behind a model comparison help determine what its result means.
- Empirical setting
- News indicators and government-bond yields
- Evidence
- Matched forecasts and controlled simulations
- Focus
- Specification choices and the scope of predictive claims
What the study finds
Positive and negative results both need context
The benchmark changes the apparent gain
Using GDELT news indicators and Eurostat yields, the application compares forecasts of volatility in a euro-area yield differential. News levels yield favourable evidence against the initial market reference; deviations from recent averages do not. Adding slower market summaries reduces the twenty-session forecast-error reduction from about 13.5% to 6.9%. None of the comparisons against the expanded reference remains significant after adjustment for multiple comparisons.
Non-detection does not settle the information question
Controlled simulations let us examine procedures when the information content is known. In the stronger-signal scenario, 58% of simulated datasets yield detection with levels but not with deviations, against the same expanded reference. This is a result within the specified simulation, showing how the chosen representation can affect detection even when additional information is present.
The empirical exercise supports claims about particular forecast comparisons. It does not establish a general causal effect of news or resolve whether news contains useful information under every reasonable specification.
An approach to interpretation
A claim–comparison protocol
The paper brings together matched comparisons, justified alternative specifications and controls with and without additional information. The aim is to connect the proposed claim to the comparison that can support it.
- State the question. Specify the informational claim before interpreting a favourable or unfavourable result.
- Align the comparison. Compare models on a common prediction task, evaluation sample and loss measure.
- Examine reasonable alternatives. Assess the roles of transformations, benchmarks and inferential choices.
- Check what the procedure detects. Use controlled settings with and without additional information, then bound the conclusion accordingly.
Abstract
Paper summary
Two competent researchers analysing the same data can write very different headlines: news improves prediction, or no improvement is detected. We examine how defensible choices produce these findings and what each permits the researcher to claim. Using GDELT news indicators and Eurostat government-bond yields, we compare forecasts of volatility in a euro-area yield differential. Matched linear forecasts yield favourable evidence for news levels, but not after subtracting recent averages. Adding prespecified slower market summaries reduces the twenty-session forecast-error reduction from about 13.5% to 6.9%; no comparison against the expanded reference remains significant after adjustment for multiple comparisons. Controlled simulations examine interpretation when information content is known. A reporting rule can credit uninformative news with market-driven gains. With the stronger added signal and the same expanded reference, 58.0% of datasets yield detection with levels but not with deviations. This paired disagreement shows why a negative result needs a check of what the procedure detects when information is present. We connect matched comparisons and justified alternatives to controls with and without additional information in a claim–comparison protocol. The method helps researchers distinguish a supported conditional gain, dependence on specification choices and an unresolved informational question.
Replication materials
Follow the evidence through the analysis
The package includes frozen inputs, analytical code, decision records, reference results and manuscript sources. It reproduces the study from the supplied inputs without requiring external data retrieval or access to AI tools.
Claude and ChatGPT supported drafting, code development and review, analysis and reproducibility checks. I directed the research and take responsibility for its substantive decisions, code, results and conclusions. The manuscript contains the full disclosure and describes the timing and limitations of the analyses.