Introducing the Salem/CSPI Forecasting Tournament

A new model for establishing academic credentials

[UPDATE, 8/13/22: Please see here for updates to the rules.]

[UPDATE, 11/17/22: Please see another update here.]

Critiques of individual experts or schools of thought have always been with us. In recent years, however, we have seen more of a frontal attack on the concept of “expertise” itself. This trend has been accelerated by the experience of the covid-19 pandemic, in which smart people across the political spectrum – conservatives, liberals, and libertarians – started paying close attention to what the public health community was doing and came away shocked by what they saw.

The argument against expertise, particularly the kind of expertise validated and legitimized by university credentials, goes something like this. Individuals succeed in business because they provide goods and services people want at the prices they are willing to pay. In academia, however, people are hired with other people’s money. There is often no objective measure of whether one’s research is any good, and individuals that do the hiring suffer no direct consequences from making bad decisions. In sum, academics become academics by writing things that people who are already academics like and agree with. This is basically the definition of a self-licking ice cream cone. When Phil Tetlock did research on whether experts were any better at forecasting events than educated laymen, he found little to indicate that they were.

The question of why expertise has gone so far off the rails has gotten a lot of attention. Diagnosing the problem of fake expertise and understanding its causes, however, are easier than providing solutions.

CSPI, in partnership with Manifold Markets and the Salem Center, wants to begin the process of trying to find one. We are announcing a program that we hope will be the first step in putting expertise on a sounder footing, by creating incentives for individuals to be correct rather than to rely on credentials, defer to authority, or conform to established ideas and ways of thinking.

The Salem Center at the University of Texas is hiring a new research fellow for the 2023-2024 academic year. This position will pay $25,000 and not require teaching or in-person residency. Rather, it will provide an academic job and financial support for a researcher to do whatever they want with their time, in order to advance their career or work on other projects.

Unlike a typical fellowship, you will not apply to this one by sending us letters of recommendation and a CV listing all of your publications and awards, and then relying on our subjective judgements about other people’s subjective judgments about your work. Rather, you will participate in a forecasting tournament on economics and social and political issues. At the end of the process, we will interview the top five finalists and decide among them. There will be a presumption towards hiring the best forecaster, but that presumption can be overcome based on how close the runner ups are and other criteria including any previous writing and business success. That’s it. You can be a high school dropout, and if you can predict what’s going to happen better than PhDs in their field, we’ll gladly make you a finalist for the job. You don’t need to share our politics, our social values, or our idea of what a scholar should look or act like.

Here’s how it works. You go to this page on Manifold Markets. After signing up, go take this survey, which asks about your name, basic demographic information, and political views for research purposes. Make sure to list your real name and username on Manifold Markets, so we can connect them to the survey results. Your answers will be kept confidential, and used only for the purposes of future research on predictors of forecasting ability among individuals.

When you sign up to play, you will get 1,000 of what we call Salem bucks (S$1,000) to distribute across markets as you wish. For example, one question asks whether Republicans will take the House of Representatives in 2022. We’ve set the price at 88 cents, corresponding to an 88% chance. Buy “yes” or “no” based on what you think will happen. You can buy based on the current price or place limit orders that depend on future price movements. For example, if you think there is an 85% chance Republicans will take the House, you can order some shares at that price, and the system will automatically buy them for you if and when it falls to that level (alternatively, as of this writing you can just buy “no” at 12 cents, since you think “no” should be at 15 cents). Gains and losses will depend on the prices at which you buy and sell shares until each market settles, at which point there will be payouts to those who own winning shares, at S$1 per share. Earnings can be reinvested. Participants will be ranked by how many Salem bucks they have accumulated by the end of the tournament on July 31, 2023.

A public leaderboard, accessible through a tab on the Salem markets page, will show who the top twenty earners are at any particular moment. If you do not finish in the top 5 of the leaderboard, there is still a chance to get something out of the market. First of all, the top 20 winners will be invited to contribute an essay to our symposium on forecasting, explaining strategies that they use and any other thoughts on the topic. They will also receive an invitation to a virtual conference on forecasting to be held at UT-Austin, which will be attended by me, Robin Hanson, Bryan Caplan, and more names to be announced later. If there is enough interest, we may make it an in-person event.

We will also give the following awards.

1. Gold-Top 1% of forecasters

2. Silver-Top 5% of forecasters

3. Bronze-Top 10% of forecasters

If you are unfamiliar with prediction markets, here’s a test page filled with nonsense questions you can bet on with Hanania bucks ($H). Once you get the hang of it, come play on the real market.

We will publicly announce the award winners, and expect that the prizes will benefit each recipient’s CV or resumé. In order to be eligible for the fellowship or any of the other rewards, participants will have to spend at least S$50 on each of 20 different markets. This will prevent anyone from winning based on one or two lucky bets.

Eventually, we hope to use the data gathered from the tournament to learn about the science of forecasting as we build upon the work of Tetlock and others.

To summarize and make things as clear as possible, the rules are once again:

1. Everyone who signs up for the Salem Center/CSPI forecasting tournament will get 1,000 Salem Bucks (S$1,000). Click here for the market.

2. Anyone can create an account and bet, but you must take this Google survey within one week of signing up in order to be eligible for prizes. All other questions are optional, but you must leave your real name and Manifold Markets username. If you want to be anonymous regarding when we publicly disclose winners and the leaderboard, that’s fine, but we’ll need your name for security purposes. Survey results will be used for research purposes but otherwise kept completely confidential.

3. Rankings are determined by wealth in S$ as of the end of the night on July 31, 2023.

4. The prizes are as follows:

   A. The top 5 individuals will be the finalists for a UT-Austin fellowship for the 2023-2024 academic year, with a grant of $25,000. Final candidates will be judged based on factors such as traditional academic criteria, business success, and public writing, with an edge given to the top forecasters.

   B. The top 20 individuals will be invited to contribute to a symposium on forecasting and a conference on forecasting to take place during fall 2023.

   C. Public honors will be given to the top 1% (Gold), 5% (Silver), and 10% (Bronze) of forecasters.

5. In order to be eligible for prizes – the fellowship, top 20 status, or a gold, silver, or bronze award – a participant will have to spend at least S$50 on each of 20 different markets. Profiles will receive a checkmark when they have met that standard. You will be able to track your current and past bets over time, so there should not be any confusion regarding what you need to do to qualify.

6. You can practice here with Hanania bucks ($H). Nothing on that particular site counts, it is simply there for practice and so you can get an intuitive understanding of the Manifold Markets system. See here for the Manifold Markets FAQ. Once again, here is the real market on which the forecasting tournament will be based. Play there when you’re ready.

7. You have to have a Google account to participate on the Manifold Markets website, and also therefore to play in the CSPI/Salem Center tournament.

8. New questions will be added regularly. Sign up to the CSPI mailing list through Substack to get regular updates.

9. I reserve the right to disqualify anyone trying to “hack” the tournament or somehow play unfairly. I don’t anticipate this happening, but there may be ways to potentially game the system that we haven’t thought of.

If you have questions, please leave them in the comments, and I’ll try to answer. Good luck!