Salem Tournament, 5 Days in

A great start, and some changes to the rules

We launched the Salem Center/CSPI forecasting tournament 5 days ago. This is the first time we – or anyone else for that matter – has done something like this, so we knew that it would be a learning process. Nonetheless, thus far we have been very excited by the results. Nearly 500 people have signed up, and close to half of them have taken the survey. Read the rules and join us if you haven’t already, and make sure to take the survey within a week of signing up to qualify for rewards.

Make sure to also follow the Salem Center account on Manifold Markets. This will give you an alert on the website whenever a new market is posted. We will also be posting new questions on the CSPI twitter account. Since we started, we’ve added two Trump indictment markets. We’ll also occasionally send out e-mails promoting recent additions.

Here are the markets that have drawn the most interest so far.

Despite our initial success, a few problems have popped up and we’ve decided to make some adjustments.

We dislike making changes to the rules, but it’s clear that this is going to have to be an iterative process, given the novelty of the tournament and what we are hoping to accomplish.

A commenter pointed to this helpful paper on some problems forecasting tournaments have as a general matter. Fiddling with the rules is a necessary evil, and we may have to do it again at some point. This is something we are going to keep being transparent about. Keep in mind that our new changes – and this will be true for any other adjustments going forward – have the goal of trying to make people reveal their true estimates of what they think is going to happen rather than strategize on how best to game the market. Those who simply bet what they truly think throughout the process will be at an advantage, and that advantage should grow over time as we figure things out.

For now, we are making the following four changes.

1. Subtracting first day winnings from the leaderboard. The first problem we saw was that there were some individuals who made a killing by taking advantage of those who did not know how the markets work (see discussion here). For example, someone came in and put all their money on China attacking Taiwan, driving it to 99%. Another participant came along and was able to take all their money. Something similar happened in the Harvard affirmative action market. The leaderboard thus came to be dominated by those who were lucky enough to have profited off of ignorance on the first day. To solve this problem, we’ve decided to subtract first day earnings from the leaderboard. So if you started out with S$1,000 and were at S$2,000 on midnight the day after launch, you’re getting S$1,000 subtracted from whatever your portfolio is now. You’re probably not hurt, and may still be at an advantage because you have more money to distribute how you see fit. It’s important to emphasize that those who made large profits on the first day did nothing wrong. For the sake of the integrity of the tournament, however, and to be fair to everyone else, we think that this is a necessary change. This problem is unlikely to happen again, as the first day was unique in bringing in a surge of inexperienced users into low liquidity markets. In the future, when a new market pops up, an alert will go out to those with experience (again, people who follow the Salem Center account on Manifold Markets and check the website), which means that we should avoid seeing wild fluctuations in price.

   

   The China attacking Taiwan market, as of about 9 hours after launching of the tournament. As you can see, YES reached 99% at one point, and 71% less than two hours later. This could not have happened with enough limit orders (see below)

2. Removing profits from the leaderboard. Some comments noted that the market incentivizes people to take long shot bets. Since being at the median doesn’t get you anything, for example, someone at the middle of the distribution should rationally take big risks in order to try to move towards the top without any fear of sinking to the bottom. This problem is quite hard to completely solve given that people are not betting their own money. What we can do, however, is limit the incentivizes to engage in risky behavior among those who are in the running for prizes. Under the old leaderboard, you knew exactly how much profit you needed to make it into the top 5 or top 20. For someone ranked 6th near the end of the tournament, it matters whether he is slightly behind 5 or way behind 5, which would imply different strategies. We have tried to deal with this problem by removing profits from the leaderboard. In other words, you can still see who the top twenty traders are, but you can’t see how wealthy they are in Salem bucks. Individuals knowing exactly what they need to do to get in the top tier leads to strategic betting, whereas the goal is to get users to bet according to their true beliefs. People not having clear insight into whether they are in the top 1%, 5%, or 10% also helps for similar reasons. Hopefully, this means anyone with a positive balance will be encouraged to bet based on their true estimates of the likelihood of events happening. We understand that this makes the leaderboard a bit less fun, but it’s probably the best thing for the tournament. We could revert to the old format later if we think the new one takes too much away from the user experience, and just live with the consequences.

3. Change in the way liquidity works. Part of the reason markets could shift so quickly early on was because they started with the default Manifold Markets setting of S$100 in liquidity. From now on, we will begin each market with the same default liquidity amount at a price we set ourselves. After 24 hours, we will add $1,900 in liquidity to stop major fluctuations in market prices, based on the idea that one person should not be able to singlehandedly cause extreme movements. This is close to what we did with the initial questions.

4. Taking into account betting behavior at the end. At the start of the tournament, we said that the top 5 leaders at the end would be judged in terms of things like academic accomplishments and business success. We are now also going to be watching for how they ended up at the top of the leaderboard. Was it a wide range of prudent bets over time, or was their success the result of a few large gambles? This will matter. We may also divide the tournament into different time periods, and factor in consistency.

Going forward, we would like to emphasize that limit orders are your friend! By placing them, you help yourself and help the market. It’s good for you because you can take advantage of other people’s foolish mistakes, and good for the market because it prevents wild fluctuations that people can take advantage of.

So let’s say that there is currently a 10% chance China strikes Taiwan according to the market. In a world with no limit orders, one foolish individual can come into the market, place everything they have on YES, and single handedly shift the probability to, let’s say, 45%. The next person who happens to come along at the right time can by pure luck make a killing by starting to buy NOs at the bargain price of 55 cents before the market shifts back. This is what happened on the first day, and we don’t anticipate it being a major problem again, for reasons mentioned above. Yet it is still something of a problem, as lesser, though still significant, irrational shifts are still theoretically possible based on the behavior of one or two people who do not understand how the markets work.

With limit orders, however, you can have people who foresaw such a thing happening, and in the case of the Taiwan question they might have placed orders to buy NO when YES hits 80 cents. Thus, the one fool cannot singlehandedly shift the market in an extreme way, and YES tops off at 20. Some people are still able to take advantage of the fool, but it’s not enough to give anyone a jackpot payout that significantly affects the outcome of the tournament.

If you have any other complaints or concerns, please don’t hesitate to reach out to us or leave comments. The feedback we have received so far has helped improve the market, and we will keep trying to make it better as time goes on.