Overview

For any random variable \(Z\) (with finite absolute first moment) and any filtration \(\mathcal F_t\) we have that \(X\) with \(X_t:=\mathbb E[Z\mid\mathcal F_t]\) is a martingale. So choosing \(Z=\mathbb{1}\{A\}\), we get that our forecasts on binary events ought to be \([0,1]\)-valued martingales.

The evolution of forecasts must satisfy the martingale property, i.e. increments must have zero mean.

This does not presume some particular model and it is data-agnostic!

While this doesn’t sound very restrictive, there is a surprisingly rich theory on \([0,1]\)-valued martingales. They need to satisfy a number of conditions “in expectation”, each of which can readily be tested empirically. This page is about

  • what (bounded) martingales can look like, and
  • finding tests for the martingale property.

Literature

Why care?

Reasoning about the future

Obviously if a forecast currently assigns a probability \(p\) to an event, we expect it to happen with probability \(p\) at some point in the future. No martingales needed for this. But a bounded martingale has other nice properties that allow us to infer more:

  1. With a current forecast of \(p_t=a\), we should expect future forecasts to never exceed \(b>a\) with probability at least \(a\over b\) via the optional stopping theorem. (E.g. for \(p_t=a=1\%,b=10\%\), the probability of this forecast ever going up to 10% again is less than 1/10=10%.
  2. More generally, for any convex, increasing function \(f\) we get that \[\mathbb P\left(\max_{s>0}f(p_{t+s})\geq \lambda\mid p_t\right) \leq {\mathbb E[f(p_\infty)\mid p_t]\over f(\lambda)}.\]
    • This follows from Doob’s martingale inequality and Jensen’s inequality.
    • Sometimes this is used for \(f\) in some family of functions such as \(f(x)=\exp(cx)\) for \(c>0\) and then minimised over \(c\) to get a tighter bound.

$$$

There is a well-documented(?) connection between Bayesian updating, holding consistent beliefs, and not being exploitable (by someone with access to the same information). [citation needed] On a high level, the intuition is that

Markets that don’t update in a Bayesian way (=prediction markets where prices are not given by martingales) leave money on the table.

At least in expectation, assuming that markets are perfectly liquid and without transaction fees, etc. This is not hard to see in principle: To violate the martingale property, the expectation of some increment has to be non-negative. If you knew this in advance, then you can buy in time and sell again after that positive EV increment realised to net a positive profit in expectation.

Can we turn this into a risk-free trade, i.e. proper arbitrage?

  • In general, not necessarily: Consider the example of a one-period binary option with payoff \($1\) if \(A\) happens and \($0\) if it doesn’t, currently trading at \(${1\over100}\), despite the fact that \(A\) is determined in the next period by a fair coin toss. Clearly the expectation of this increment \({99\over100}-{1\over100}\over2\) is not 0, so this is in clear violation of a martingale. Yet we cannot gain money (with positive probability) without also potentially losing some; no matter how we short/long things.

  • In some cases, however, e.g. if the quadratic variation is sufficiently high, you should be able to guarantee the existence of swings wide enough that you can buy low, sell high, as often as you wish.

Note that these martingale violations are not necessarily easy to spot. For example, some forecasts look like they are “clearly not martingales” while actually (potentially) being martingales, while others may look reasonable, but actually fluctuate way too much.

How to spot martingale violations

Intrinsic time

I found it useful to think in terms of a martingale’s “intrinsic time”. This is analogous to the reparameterisation needed for the Dubins-Schwarz Theorem which says that every (local) continuous martingale with diverging quadratic variation is just a reparameterised Brownian motion. (Unfortunately, this is not directly applicable in a forecasting context, since certain news will make forecasts jump, hence violating the continuity assumption, and because every \([0,1]\)-valued martingale has finite expected quadratic variation ≤0.25 – see the section below.)

Intuitively, this just means that we do not care whether information is revealed once in a hundred years and then rapidly in succession within three days or more evenly distributed. This also hints at the fact that it is intrinsically hard to tell apart (on a path-wise level) martingales with low quadratic variation from non-martingales, or equivalently, forecast trajectories for events where we do not get a lot of information over time to update on.

Examples of prediction martingales

The biggest obstacle to detect martingale violations is that martingales come in lots of different forms and shapes; hence statistical tests for martingale detection that work in such generality should not give false negatives for a surprisingly large set of processes. I found it useful to keep the following examples in mind:

Will \(X\) happen?” with lots of minor updates on \(X\)

The “typical” example of a forecast trajectory. Since updates are assumed to be minor and frequent, this will have a decidedly non-zero (but not diverging!) quadratic variation \(\sum_t(p_t-p_{t-1})^2\) and (at least for a given discretisation) diverging total variation \(\sum_t|p_t-p_{t-1}|\). Hence its paths will be the most similar to Brownian motion, but of course it is not Brownian motion: There can’t be as many swings and obviously it’s bounded in \([0,1]\).

Data of the evolution of a forecast with potential excess volatility from here via @aubreyclayton’s statmodeling post:

Will \(X\) happen? with few major updates on \(X\)

Trajectories will be piecewise constant with several jumps. Slightly annoying if you planned to invoke any results for continuous martingales.

“Popular” results for continuous martingales include:

  • Stopping times like the first time \(x\) is hit are comparable for different \(x\).
    • See e.g. here, where it is used to derive test statistics involving the number of crossings and the distribution of the minimum, conditional on the event resolving positively.
  • Any continuous martingale with finite total variation is constant almost surely.

One such example is the evolution of the Metaculus community’s estimates of whether Trump will be back on Twitter in 2022:

image.png which is essentially constant up to one small bump when Musk started hinting at buying Twitter and one big bump when Musk bought Twitter.

Will \(X\) happen? without updates on \(X\)

Trajectories will start from some point in \((0,1)\) and essentially stay constant since, by assumption, there is hardly any new information to update on/to make predictors deviate from their base-rate estimates. This is an example where a lot of time may pass, but the intrinsic time is effectively 0. This can mess up some statistics if they are trying to detect “too much movement” among an ensemble of realised trajectories.

An example for this would be the above question about Trump’s Twitter account if Musk had not done anything.

Will an exponential clock tick before \(t\)?

A weird one because sample paths with go down almost linearly unless it suddenly jumps to 1 and stays there. A definite counterexample to one can’t anticipate the direction of movement.

One such example might be the community median on whether Kyiv would fall to Russian forces by a certain date:

image.png

It didn’t take long for people to start criticising the Metaculus forecast (ACX/SSC), as arbitrage opportunities obviously show that this can’t be a Bayesian forecast, etc. (Spoiler: It can be. At the very least we cannot reject such a forecast as un-Bayesian on martingale/hypothetical betting market considerations alone.)

It should also be mentioned that we can get all sorts of funny “typical trajectories” by introducing uncertainty about the rate of our exponential clocks, as outlined here.

A perfect exponential clock (without noise by taking the median forecast or clearing price, etc.) can be formulated as follows and is handy to keep in mind: Given an exponential clock \(C\) with parameter \(1\), i.e. \(\mathbb P(C\leq t)=1-e^{-t}\), and the event \(A={C\leq T}\), we have \[p_t = \mathbb P(A\mid \mathcal F_t) = \mathbb 1(C\leq t)+ \mathbb 1(C>t)\underbrace{\mathbb P(C\leq T\mid C>t)}_{1-e^{-(T-t)}}.\] Let us sample a couple of these paths:

Test #1: Excess volatility

Augenblick & Rabin define

  • \(u_t := p_t(1-p_t)\), the uncertainty at time \(t\),
  • \(r_{t_1,t_2}:=p_{t_1}(1-p_{t_1}) - p_{t_2}(1-p_{t_2})\), the uncertainty reduction between two periods \(t_1,t_2\), and
  • \(m_{t_1,t_2} := \sum_{\tau=t_1}^{t_2-1}(p_{\tau+1}-p_\tau)^2\), the movement across those periods

for discrete-time forecasts \((p_t),t\in\mathbb N\). Note that \(m_{0,t}\) is simply the quadratic variation \([p]_t\) of the process.

Consider now the one-period excess movement statistic \[m_{t,t+1}-r_{t,t+1} = (p_{t+1}-p_t)^2-p_t(1-p_t)+p_{t+1}(1-p_{t+1}) = (2p_t-1)(p_t-p_{t+1}),\] which is a centered random variable if $p_t$ is a martingale. By linearity of expectation, this means that the expected total uncertainty reduction equals the expected total movement (i.e. the expected quadratic variation).

Psychological biases may vaguely be classified as follows:

  • Base-rate neglect and overreaction always lead to expected movement > expected uncertainty reduction, while
  • both confirmation bias and underreaction lead to expected movement < expected uncertainty reduction.
  • Base-rate neglect can be seen as expected movement > expected Bayesian movement and expected uncertainty reduction < expected Bayesian uncertainty reduction. (I.e. we move more and end up more uncertain, on average.)
  • Confirmation bias (if strong enough) is expected movement < expected Bayesian movement (too little movement) and expected uncertainty reduction > expected Bayesian uncertainty reduction (overconfident).
  • Overreaction is expected movement > expected Bayesian movement and expected uncertainty reduction > expected Bayesian uncertainty reduction,
  • while underreaction is expected movement < expected Bayesian movement and expected uncertainty reduction < expected Bayesian uncertainty reduction

Example: Did 538 update too much?

Let us look at the data from the “lots of minor updates on \(X\)” example:

Since the one-period excess movement statistic has zero mean, we have that the expected quadratic variation \(\mathbb E[p]_\infty = m_{0,\infty}\) equals the expected uncertainty reduction \(\mathbb E r_{0,\infty}=\mathbb E[p_0(1-p_0)-p_\infty(1-p_\infty)]\leq {1\over4}\). Markov’s inequality thus immediately gives \[\mathbb P([p]_\infty > \lambda )\leq {\mathbb E[p]_\infty\over \lambda}\leq {1\over4\lambda}.\]

The quadratic variation above is roughly 1, so the probability of this sample path coming from a martingale is less than 25%.

In fact, the paper goes further by bounding the variance of the quadratic variation under a martingale, giving the bound

\[\mathbb P\left([p]_\infty > u_0+\sqrt{u_0\left({3\over2}-u_0\right)}{\sqrt{1-\delta}\over\sqrt\delta}\right)\leq \delta,\]

where \(u_0=p_0(1-p_0)\). This is better than the naive Markov bound only for more extreme values of \([p]_\infty\) (i.e. smaller values of \(\delta\); roughly \(\delta <{1\over6}\)) and does not improve upon the naive Markov bound in our example at hand.

Note, however, that none of these bounds are really that strong. Considering that the forecasts for the previous two elections were apparently significantly less volatile, I’d say that this is actually well within expectations.

Certainly not strong enough to dismiss the forecast as “definitely non-Bayesian” and even weaker than the fact that we went from >95% back to 0% – by definition this happens with probability <5%! But in hindsight we can always find events of (a priori) arbitrarily small probability that were realised.

What about other tests like autocorrelation of increments?

(Nothing to see here either.)

Test #2: Autocorrelation tests

Note that for every martingale \(X_t\) with increments \(D_t=X_t-X_{t-1}\) we have \[\forall s\neq t:\quad \mathbb E[D_sD_t]=0,\] i.e. increments are uncorrelated. (Clearly increments have zero mean, so the above is the covariance.To see that this holds, we assume without loss of generality that $s<t$ and expand using the tower property: \(\mathbb E[D_sD_t] = \mathbb E[D_s\underbrace{\mathbb E[D_t\mid\mathcal F_s]}_{=0}]\).)

This gives another test statistic. It suffers from several drawbacks in comparison to the excess movement test above, however:

  • It is not invariant under reparameterising time, i.e. not a function of “intrinsic time”.
  • It gives rise to a huge amount of test statistics and thus a certain risk of \(p\)-hacking.

More general tests via Itô’s lemma

In both the excess movement test and the autocorrelation test, we used a decomposition à la \[\mathbb E\left[f\big((p_s){s\leq t})\big)(p{t+1}-p_t)\right] = 0,\] by the martingale property. (Different choices of \(f\) give either of the tests.) Now we might plug in various \(f\) above to get all sorts of tests!

First, however, let us rephrase the above in a more natural continuous-time setting which gives us access to handy tools from stochastic calculus.

A brief “dictionary”:

  • \(p_{t+1}-p_t \Leftrightarrow dp_t\)
    • \((p_{t+1}-p_t)^2 \Leftrightarrow dp_tdp_t = d[p]_t\), where \([\cdot]_t\) denotes the quadratic variation
  • \(f({p_s}_{s\leq t},t) \Leftrightarrow X_t\), where \(X_t\) is some \(\mathcal F_t\)-adapted process.

Itô’s lemma

Essentially this is a glorified Taylor expansion, highlighting that stochastic second order effects are potentially of the same order of magnitude as first order effects. Think of Brownian motion where \(dB_t\approx B_{t+dt}-B_t = \mathcal N(0,\sigma^2=dt)\).

Lemma: For \(f\in C^2:x=(x^1,\dots,x^n)\mapsto f(x)\in\mathbb R\) we have \[df(X_t) = \sum_{i=1}^n {\partial f\over\partial x^i}(X_t)dX^i_t + {1\over2}\sum_{i,j=1}^n {\partial^2 f\over\partial X^i\partial X^j}(X_t)\underbrace{dX^i_tdX^j_t}_{=d[X^i,X^j]_t}.\]

Tests

Now note that, as above, for every adapted \(X_t\) and every martingale \(p_t\) we have \[\mathbb E\left[\int_0^T X_tdp_t\right]=0.\] Note also that \(\int_0^T X_tdp_t\) can be seen as the (random) payoff of a continuously rebalanced portfolio holding \(X_t\) units of some contract worth \(p_t\) at time \(t\). So the fact that the expectation vanishes just tells us that there is no betting strategy with positive expected value when playing fair games.

Now let us see what kind of tests we get for different choices of \(X_t\):

  • \(X_t=1\) is a weak test for calibration: \[\int_0^T dp_t = p_T-p_0,\] so if we consistently start with an estimate \(p_0\) that is strictly less, say, than what we estimate after learning more (i.e. our probability \(p_T\) at time \(T\); for \(T\) large enough we may assume \(p_T\in{0,1}\), i.e. the question was resolved). Note, however, that initially guessing too high on some questions and consistently guessing too low on others will cancel out.
  • \(X_t=\mathbb 1\{p_0\in [a,b]\}\) gives \[\int_0^T\mathbb 1\{p_0\in [a,b]\}dp_t = \mathbb 1\{p_0\in[a,b]\}(p_T-p_0),\] a weak calibration test, conditional on the initial forecast being in \([a,b]\).
  • \(X_t=p_t\) gives \[\int_0^T p_tdp_t = {1\over2}(p_T^2-p_0^2 - [p]_T),\] which doesn’t seem too useful on its own.
  • \(X_t=1-2p_t\), however, just gives Augenblick&Rabin’s excess volatility test: \[\int_0^T (1-2p_t)dp_t = [p]_T+p_T(1-p_T)-p_0(1-p_0).\]
  • \(X_t=-\log\left({p_t\over1-p_t}\right)\) gives \[-\int_0^T \log\left({p_t\over1-p_t}\right)dp_t = \log\left[p_0^{p_0}(1-p_0)^{1-p_0}\right]+{1\over2}\int_0^T{1\over p_t(1-p_t)}d[p]_t,\] where the integrand is negative for \(p_t>{1\over2}\) and positive for \(p_t<{1\over2}\), so it is another test of “mean reversion”/excess volatility (referred to as Kullback-Leibler test in Augenblick&Rabin).
    • This can be seen by interpreting the l.h.s. as \(\int_0^Tf’(p_t)dp_t\) with \(f’(x)=-\log\left({x\over 1-x}\right)\); thus \(f(x)=-\log\left(x^x(1-x)^{1-x}\right)\) and \(f’‘(x)=-{1\over x(1-x)}\).

    • By Itô, we have \[f(p_T)-f(p_0) = \int_0^Td(f(p_t)) =\int_0^T f’(p_t)dp_t + {1\over2}\int_0^T f’‘(p_t)d[p]_t.\]

    • Plugging in the expressions for \(f,f’,f’’\), noticing that \(f(0)=f(1)=0\) (we assumed \(p_T\in\{0,1\}\)), and rearranging gives the desired claim. (Recall that \(d[p]_t = (p_{t+1}-p_t)^2\) in our discrete world.)
  • \(X_t = \mathbb 1\{p_t\in[a,b]\}\) gives (for continuous \(p_t\)) \[\int_0^T\mathbb 1\{p_t\in[a,b]\}dp_t =\]
    • \(b-\max(a,p_0) \mbox{ if } p_0\leq b,p_T=1\)
    • \(a-\min(b,p_0) \mbox{ if } p_0\geq a,p_T=0\)
    • \(0 \mbox{ otherwise.}\)

We should be able to obtain tests for all sorts of other martingale properties like crossing inequalities, Doob’s martingale inequality, etc. by choosing different \(X_t\)s.

Making $$$ off martingale violations

Some claimed that the above forecasting martingale fluctuated too much and thus would be susceptible to (statistical) arbitrage, i.e. positive expected value bets.

Note that \(1-2p_t\), the integrand of the integral that gives rise to the Augenblick & Rabin test statistic \([p]_T-p_0(1-p_0)\), is positive if and only if

  • \(p_t>{1\over2}\) and the probability decreases (i.e. \(p_t>p_{t+1}\)), or
  • \(p_t<{1\over2}\) and the probability increases.

So if this integral is positive in expectation (while it should really vanish), this suggests that probabilities tend to revert back towards \(1\over2\) more than they should. This suggests either base-rate neglect or overreacting to new information (as defined in Augenblick&Rabin). Intuitively, we should be able to make money off this observation by “buying low and selling high”. But keep in mind, that we could keep accumulating (excess) quadratic variation simply by jumping back and forth between \(x\)% and \(x+1\)%, so merely buying at a previously determined lower bound (say 10%, i.e. \(p=0.1\)) and selling at a previously determined upper bound (say 90%) won’t work in general.

Instead, our intuition is simply that (if we expect the test statistic \([p]_T-p_0(1-p_0)\) to be positive) we should go long if \(p_t<0.5\) and short if \(p_t>0.5\). Now we will make this intuition more precise by generalising a bit:

Note that stochastic integrals of the form \[\int_0^T f(t,(p_s)_{s\leq t})dp_t \approx \sum_{t=0}^T f(t,(p_s)_{s\leq t})(p_{t+1}-p_t)\] can be seen as the payoff of an investment strategy where at time \(t\) we buy (/sell) \(\mathrm d\!f\) binary contracts. For example, choosing \(f=2\times \mathbb 1(t\in[a,b])\) corresponds to buying two contracts at time \(t=a\) and selling them again at time \(t=b\) for a profit (/loss) of \(2\sum_{t=a}^{b-1} (p_{t+1}-p_t) = 2(p_b-p_a)\). By linearity, this idea immediately extends to all step functions and their closure in whatever topology you want to consider.

Now one obvious choice for \(f\) that exactly replicates the payoff we assumed to have positive expectation is \[f(t,(p_s)_{s\leq t}) = 1-2p_t,\] so that we get (assuming that the event resolved by time \(T\), i.e. \(p_T\in{0,1}\)) \[\int fdp_t\approx \sum_{t=0}^T (1-2p_t)(p_{t+1}-p_t) = \sum_{t=0}^T[m_{t,t+1}-r_{t,t+1}]=[p]_T-p_0(1-p_0).\]

This is slightly different from the trade suggested by Taleb&Madeka, which is given by

\[f(t,(p_s)_{s\leq t}) = 2(p_0-p_t),\]

with payout

\[\int fdp_t\approx -2\sum_{t=0}^T \left(\sum_{s=1}^t\Delta_s\right)\Delta_{t+1} = -2\sum_{0<i<j\leq T}\Delta_i\Delta_j = -\sum_{i\neq j}\Delta_i\Delta_j = \sum_i\Delta_i^2-\underbrace{\sum_{i,j}\Delta_i\Delta_j}_{\left(\sum_i\Delta_i\right)^2} = [p]_T-(p_T-p_0)^2,\] where we defined \(\Delta_s = p_s-p_{s-1}\).

More directly via Itô’s lemma: Idea: Want quadratic variation to enter the picture, so we’ll need something with non-vanishing second derivative to apply Itô’s lemma to:

  • \(d(p_t)=dp_t\),
  • \(d(p_t^2) = 2p_t dp_t + d[p]_t\),
  • ⇒ \(d(p_t-p_t^2)+d[p]_t = (1-2p_t)dp_t\)
  • ⇒ \(\int_0^T(1-2p_t)dp_t = \underbrace{p_T(1-p_T)}_{=0}-p_0(1-p_0) + [p]_T\).

Quite generally, this leads to a handy framework for constructing trading strategies in general: If we suspected the \(k\)-autocorrelation to be positive, for example, we should hold \(f(t,(p_s)_{s\leq t})=p_{t-(k-1)}-p_{t-k}\) assets at time \(t\).

(Naively) correcting the market doesn’t necessarily give better forecasts

Admittedly a trivial observation: Imagine Warren Buffett realised that some prediction market was too volatile and started placing huge bets according to one of the strategies discussed above, thus moving the market price. The resulting evolution of market prices certainly exhibits less of an edge for others who also observe the excess volatility problem (but can’t trade before Buffett places his orders), so in this sense these prices are “better”. Unfortunately, a market price moved like this may now exhibit other “defects”, e.g.

  • autocorrelation if excess volatility was “corrected” via rolling averages or
  • underreacting to (some) novel information if excess volatility was “corrected” via dampening of probabilities (moving them closer to 50:50).

In terms of log-score, naive dampening in the presence of excess volatility seems to be bad (as evidenced by numerical simulations of Metaculus data here).

Fixing this seems hard since we “trade” having an easily detectable flaw in our model (a martingale violation which is data and model independent) for introducing all sorts of other flaws which are typically harder to correct.