8th Out of 34618 In Public Metaculus Predictions

by

I have made several thousand public predictions at Metaculus. I , Brian Wang, ranked 8th in Baseline Accuracy out of 34617 forecasters. This is a 99.98 percentile ranking.

Apparently, some people do not understand a description converted into a mathematical term. Percentile is a mathematical term that is not saying there was X% correct. I was about 98%+ correct.

Perplexity explains:

Metaculus is an online forecasting platform and aggregation engine.

The Peer Accuracy score is a new metric they created in the last year or so and retroactively applied. The Peer Accuracy metric has a time component where if you are late entering a prediction because you did not see it, then you can get less points despite a perfectly correct prediction. A question is open for five years and out of 17000 questions, it does not catch a forecasters attention until year 4. A prediction is entered for the final year and it is accurate. The points for that question get reduced to only 20% of possible because you were in the question for 20% of the life of the question.

Baseline accuracy is points based upon how right the predictions were and the number of right predictions with losses for being wrong.

20 thoughts on “8th Out of 34618 In Public Metaculus Predictions”

  1. I am the dude in China thats left a few comments before.

    Brian, I think highly of you and your predictions. 3 questions.

    1. You discuss the scoring mechanisms. Can you confirm you are not gaming these mechanisms to inflate your score?

    2. A prime question in your lane is the Metacalc question of “Will Tesla land humans on mars before 2030.” Have you answered this? I havent been able to see your response.

    3. What is your answer to #2? It might be the defining prediction of our age. Could you provide a brief answer here? I recently am having a hard time loading Metacalc.

    • I only spend a little time making predictions on Metaculus. There is virtually no money in it for me. I did win a little bit in a tournament. I won more money Manifold.

      It is not worth my time to spend time gaming Metaculus.

      My main effort is reading question specifics quickly and determining if it is a pitch worth swinging at. Will I know what will happen with certainty and are there no gotchas with the question.

      I will make a quick post on humans on mars and my predictions of it.

  2. I will try to be objective and look for baseline accuracy for last 5 years and rankings 1 – 400.
    Nextbigfuture made 507 predictions so it is fair that I put it in the middle and include people who made 250 – 750 predictions and not others.

    https://www.metaculus.com/leaderboard/?category=baseline&year=2021&duration=5
    So Nextbigfuture ranks about 13h out of 44. That is (13/44) * 100 = 29.5%, so not bad, but not the best, about better than 70.5 % and worse than 29.5 %.

    I think that peer accuracy awards people making the most predictions. So if you make more than 50 % right you get more positive points than bad and climb up. The first guy is the one who made the most predictions by far and did not suck too much at it and the top are the persons with the most predictions. After 100th person you rarely see anyone making 200+ predictions,..

    So I cant buy that 99.8 % claim. Better than 70.5 % yes or a bit more and that is it.

  6. I think 99.8 % is a bit misleading. https://imgbb.com/ I am not into statistics, but doesnt this metric show many false predictions?

    Number of gigafactories was false prediction, Tesla stock value has some good and some false predictions. Lk 99 has many false predictions,.. Autopilot dates have lots of false predictions.

    • It’s not misleading. Brian is 99.8th percentile on that platform. That’s true. And it means he really does make better predictions than almost anyone there.

      News coverage of LK99 is not a prediction. This number is about formally making a measurable and testable prediction on that platform.

      This really is impressive. Congrats, Brian!

      • Yeah that sounds nice and congrats, but how many of them if one of few times forecasters, who just registered and tried and then gave up? I guess majority.

        1 – 100 ranked are the ones who answered about 100+ questions.
        Later on the curve goes does down. I guess the majority answered only a few questions and so they are ranked badly. Algorithm probably favors the people who answer the most questions, so they are on top.

        I am more interested about how many % of predictions were true and how many false, but that site wont tell you that.

      • Sorry I can not buy that crap.

        https://postimg.cc/k23ctHxx

        Even at positions about 400 you see people, who had made only 20 – 30 predictions and not the ones making more of them. By the time we come to 2000 – 30000 we will see people ranked, who made only a few predictions and or just made a registration on website. Should we count them? Strangely the website wont show complete rankings why not? Yeah but by the sheer number of predictions made nextbigfuture ranks very high, like others who made similar number of predictions, algorithm favors the ones answering the most and puts them on top. I am not saying Brian doesnt make some good predictions and or course some bad ones. I am saying better than 99.8 % is misleading and according to data I do not buy it.

    • 99.98 percentile is how 8 out of 34617 converts to percentile.

      It is math.

      AI explains it.
      Being 8th out of 34,617 forecasters in Baseline Accuracy on Metaculus is an exceptionally high ranking. To express this as a percentile, we can calculate it as follows:

      Calculate the percentile rank:
      Percentile Rank = (34617 – 8) / 34617 * 100 = 99.9769%
      Round to two decimal places:
      99.98th percentile

      This means that the performance is better than 99.98% of all other forecasters on the platform. To put this into perspective:

      It places the forecaster in the top 0.02% of all participants.
      Only 7 other forecasters out of 34,617 performed better.
      This level of accuracy is extremely rare and indicates exceptional predictive skills.

      Such a high percentile ranking demonstrates a remarkable ability to make accurate predictions across various topics on the Metaculus platform.

      • That is nice, but majority of forecasters are probably just registered and answered a few questions as I looked into it. So it is not objective and algorithm probably favors the ones answering the most questions.

        I looked into last 5 years, 2021 – 2025.

        So if we eliminate the ones who are not serious, we see that about top 50 answered 200 + questions from which Nextbigfuture is 15th. So about in top 70% percent but not 99.8 %.
        Then the curve goes down, and about top 120 answered more than 100 questions. Nextbigfuture is then in top 10 percent, although the algorithm clearly favors the ones answering the most. From 126 – 500 I say only 1 person answering more than 200 questions. Lots of them only answered 20 – 70 questions. Then the site wont show you the curve, but I guess the ones ranking at the bottom made only a few predictions and gave up, or were not serious. So that 99.8% score is not accurate for my standards, since algorithm clearly favors the ones answering the most and puts them on top.

        From the ones, who answered 100 questions or more Nextbigfuture is better than about 90 %, or from 200+ questions answered better than 70%.

        • There are plenty of forecasters below who were in for 300 or more questions. Plus those who were in for 50-200 questions did not have half of my score. Several of the ones above me 600+ questions answered.

          in the 5 year I am ranked 15th but the metaculus ranking algorithm gives me 8th overall. You can parse and make adjusted algorithms.

          I can also say that I had ten or so questions that turned badly with -150 to -400 some points. Out of the 500 or more questions. So 2% of the time, I am particularly off. But I can confidently direct and make the right call 97% of the time. This means months or even years earlier on typical questions, I am getting enough info to know what will happen and I can tell people. Not perfect but the vast majority of the time.

          • If you want to say you are better than the ones who just registered and made 1 or few predictions be my guest, I dont count that.

            For baseline accuracy, last 5 years: From rank 100 to 400 (website wont show more) I counted only 5 people making more than 200 predictions.

            • Kareem had the most career points before Lebron. He played many years. He was averaging 20-30 points per game. People who never got 24 points per game and played too few games are obviously not better. Metaculus runs there site and creates there algorithm. They rank me 8th out of 34617 on baseline accuracy. Their metric and their ranking. How do you adjust Kareem? He’s behind Magic and Bird on career points because Magic and Bird would have gotten more theoretical points if they had played more games? Or how about Brandon Roy or Grant Hill? Kareem average over 24 points per game for 20 years.

              I averaged 34.6 points for each of the 517 questions. I had to pick not just my 50 most certain questions but 467 others. I had to not just take layups but went for more shots. Some of the people down the list who had 30 or 60 questions. They picked a bunch of obvious election picks or they did some sports questions. They could not find more questions that they had confidence to predict. They would not as done as well on low confidence and less known areas of knowledge.

          • Example Vang vs Me.
            Me (517 total, 45 wrong less than random, 17 way wrong with over -100. one of them was the LK99, 7 way way wrong)
            https://www.metaculus.com/contributions/baseline/110947/?year=2021&duration=5

            Vang (535 total, 100 wrong less than random, 15 way wrong less than -100, 3 way way wrong -200 or less)
            https://www.metaculus.com/contributions/baseline/111881/?year=2021&duration=5

            At the bottom you see the red questions where I was wrong and red questions where he was wrong.

            Where was I way wrong? I thought Trump would get over 10 million followers on another social media platorm. It ended being about 8.2 million at Nov 2024 Truth social for the resolution date. Trump now has 8.9 million followers on Truthsocial. Instagram: 31.7 million followers, 100 million on X. But Instagram and X and others did not count

            According to the most recent data available, Truth Social had approximately 25.9 million monthly website visits in January 2025. And growing about 10-20% per month.

            I did not think OpenAI would create a board risk committee by jan 1 2025 and they did.
            I thought the US would enter a recession which I think did happen in 2023 but statistics were manipulated so it was not official
            I was too late thinking the Celtics would win NBA finals 2024.
            I was too confident thinking the houston texans would not make the divisional playoffs
            I thought Israel would get control of Gaza by the terms were Israel has clear military control of a majority of Gaza City by area and does not recognize any governing authority.on Jan 1, 2025.
            There were other terms about over 50% of Gaza territory being contested. I disagree with the resolution of this question.

            • Nextbigfuture made 507 predictions so it is fair that I put it in the middle and include people who made 250 – 750 predictions and not others. Look at rankings 1 – 400. Even after 100th there is rarely one making 200+ predictions and the curve goes down so we wont included the ones who just tried the site and made only a few predictions. Even if they made 10 accurate ones they are still ranked way behind, since number of predictions has lots of influence and not only accuracy.

              https://www.metaculus.com/leaderboard/?category=baseline&year=2021&duration=5
              So Nextbigfuture ranks about 13h out of 44. That is (13/44) * 100 = 29.5%, so not bad, but not the best, about better than 70.5 % and worse than 29.5 %.

              So I cant buy that 99.8 % claim. Better than 70.5 % yes or a bit more and that is it.

Comments are closed.