Great writeup, very interesting results! Congrats on completing it, and congrats to the winners.
I work at Manifold and helped run the tournament. Was surprised that political leaning was so important to the results!
The final calibration was definitely underwhelming, but I think it's not all due to the effect of people trying to take risky bets to win the top prizes. Another factor was a limited supply of currency with which to bet.
One condition of the tournament was that everyone began with a fixed amount of currency: S$ 1000. Each market has an AMM (automated market maker) which was also started with about S$ 2000 liquidity. As a player, you might not want to spend all your balance moving the probability from 85% to 95% if you could make more profit in another market. On the main Manifold site, you can buy more currency, and also there are automatic loans that return you your currency to bet again, which both help.
This is kinda true, but if you look at https://calibration.city/manifold the calibration curves for the main site also exhibit this S curve. Some hypotheses:
- Mana isn't redeemable for real money, so there's still a limited-balance problem just not as bad. Test by looking at polymarket's calibration curves, or by trying even looser monetary policy (bad idea?)
- Limit orders are more granular near the middle than the tails, because we require them to be integer percents. Test by allowing more limit order precision. I'm skeptical of this one - Jack doesn't think it matters much.
- The AMM itself is wrong
- This problem intrinsically exists in all prediction markets because other investments are more attractive than pushing a market closer to 100% or 0%. I like this reason the most. Counterpoint: Nuño's "resolves YES" markets are all close to 100% despite being far in the future https://manifold.markets/Nu%C3%B1oSempere?tab=questions&q=resolves+positively
What's your explanation for why, aside from the midterms, conservatives did better? Was there some event where Republicans did surprisingly well which you should also exclude?
Not sure what else we would exclude, as nothing was like elections in being likely to lead people in a partisan direction. Elections were also the biggest individual category, nothing else with that many markets naturally clusters together. But if someone has a theory we could test it.
As someone who actually participated in the contest and thus saw how things worked first-hand, your analysis in the first section is *really bad*.
First off, your list of "biggest misses" makes no sense. The actual biggest miss was the Debt Ceiling market, which peaked at 93%, but there were tons of other misses larger than the ones you listed (off the top of my head, China GDP, all the major midterm markets, China COVID, Biden Approval, Erdogan, and Trump Indictment, and probably others as well.)
Additionally, your analysis of the tail probability effect is *way* off the mark. While it is true that there were a few cases of people gambling on longshot odds, that wasn't a significant factor. The main reason that even sure-thing markets would not go to 0/100 is because the participants never had enough liquidity to push them up that much.
The YES/NO asymmetry is probably due to differences in resolution timing affecting the opportunity costs, because most of the questions were posed as "Will X happen before Y time?". This means that betting on NO involves locking up your money until the given date, which was usually months in the future, and thus there is a huge opportunity cost to betting NO. Betting YES meanwhile had no such opportunity cost.
It was a fun contest, though I realized I stood no chance of winning once the china COVID market closed, I really missed the ball on that one. I'm surprised I ended up top 10 even after quitting 2/3rds of the way through.
A shorter one has its own problems because it could end up all just being luck. That's a problem here, but the shorter it is the worse the problem is. Or you won't have enough markets on important things.
Would seem that the better idea is to maximize the diversity of questions to make performance on each as uncorrelated as possible. This prevents a single larger episode (Ukraine war, midterms) from affecting results so much.
Great writeup, very interesting results! Congrats on completing it, and congrats to the winners.
I work at Manifold and helped run the tournament. Was surprised that political leaning was so important to the results!
The final calibration was definitely underwhelming, but I think it's not all due to the effect of people trying to take risky bets to win the top prizes. Another factor was a limited supply of currency with which to bet.
One condition of the tournament was that everyone began with a fixed amount of currency: S$ 1000. Each market has an AMM (automated market maker) which was also started with about S$ 2000 liquidity. As a player, you might not want to spend all your balance moving the probability from 85% to 95% if you could make more profit in another market. On the main Manifold site, you can buy more currency, and also there are automatic loans that return you your currency to bet again, which both help.
See also "Opportunity costs are critical" in this writeup: https://blog.polybdenum.com/2023/08/01/how-i-came-second-out-of-999-in-the-salem-center-prediction-market-tournament-without-knowing-anything-about-prediction-markets-and-what-i-learned-along-the-way-part-1.html
The effect of limited currency meant users did not bet probabilities as far as they would have, which matches the worse calibration at the extremes.
This is kinda true, but if you look at https://calibration.city/manifold the calibration curves for the main site also exhibit this S curve. Some hypotheses:
- Mana isn't redeemable for real money, so there's still a limited-balance problem just not as bad. Test by looking at polymarket's calibration curves, or by trying even looser monetary policy (bad idea?)
- Limit orders are more granular near the middle than the tails, because we require them to be integer percents. Test by allowing more limit order precision. I'm skeptical of this one - Jack doesn't think it matters much.
- The AMM itself is wrong
- This problem intrinsically exists in all prediction markets because other investments are more attractive than pushing a market closer to 100% or 0%. I like this reason the most. Counterpoint: Nuño's "resolves YES" markets are all close to 100% despite being far in the future https://manifold.markets/Nu%C3%B1oSempere?tab=questions&q=resolves+positively
What's your explanation for why, aside from the midterms, conservatives did better? Was there some event where Republicans did surprisingly well which you should also exclude?
God was with them? I don't know.
Not sure what else we would exclude, as nothing was like elections in being likely to lead people in a partisan direction. Elections were also the biggest individual category, nothing else with that many markets naturally clusters together. But if someone has a theory we could test it.
As someone who actually participated in the contest and thus saw how things worked first-hand, your analysis in the first section is *really bad*.
First off, your list of "biggest misses" makes no sense. The actual biggest miss was the Debt Ceiling market, which peaked at 93%, but there were tons of other misses larger than the ones you listed (off the top of my head, China GDP, all the major midterm markets, China COVID, Biden Approval, Erdogan, and Trump Indictment, and probably others as well.)
Additionally, your analysis of the tail probability effect is *way* off the mark. While it is true that there were a few cases of people gambling on longshot odds, that wasn't a significant factor. The main reason that even sure-thing markets would not go to 0/100 is because the participants never had enough liquidity to push them up that much.
The YES/NO asymmetry is probably due to differences in resolution timing affecting the opportunity costs, because most of the questions were posed as "Will X happen before Y time?". This means that betting on NO involves locking up your money until the given date, which was usually months in the future, and thus there is a huge opportunity cost to betting NO. Betting YES meanwhile had no such opportunity cost.
Which occupations were the most successful at forecasting?
It was a fun contest, though I realized I stood no chance of winning once the china COVID market closed, I really missed the ball on that one. I'm surprised I ended up top 10 even after quitting 2/3rds of the way through.
Maybe a shorter timespan contest next time?
A shorter one has its own problems because it could end up all just being luck. That's a problem here, but the shorter it is the worse the problem is. Or you won't have enough markets on important things.
Would seem that the better idea is to maximize the diversity of questions to make performance on each as uncorrelated as possible. This prevents a single larger episode (Ukraine war, midterms) from affecting results so much.