AI Have Years of Failure then Stunning Success

There are many who believe that years of failure or limited success for Tesla FSD and other robotaxi projects are evidence for their belief that it will take more years from today for Tesla FSD to achieve robotaxi level.

Chess Computer Programs

Chess programs were under serious development for decades before they finally succeeded in beating the human chess grandmaster.

In 1978 a chess engine named Belle won the North American Computer Chess Championship run by the Association for Computing Machinery, the engine’s special hardware allowed it to analyze around thirty million positions in three minutes. Belle also held both opening and ending database’s, greatly aiding the hardware speed. In 1980, Belle became to first chess engine to receive a Master rating.

In 1995, a new chess engine prototype was released from the team at IBM, Deep Blue. In 1996, Deep Blue faced chess champion Garry Kasparov for the first time. Kasparov won the six-game match by the score 4–2. In 1997, Deep Blue beat Garry Kasparov.

History of Computer Go

From 1971 to 2003, the Computer Go programs could barely play and would lose badly to human amateurs despite the Computers getting large handicaps.

Competitive Without Handicap with a Good Amateur

There was significant improvement from 2004 to 2014. In 2014, the software Crazy Stone and eleven-times German Go champion Franz-Jozef Dickhut, 6 dan amateur, played without a handicap. Dickhut won as was expected by most observers and himself before the match. However Crazy Stone won the first game by 1.5 points, which was a resounding mark that the top programs have reached top amateur level. In 2015, Dickhut won again 3-1 with computer program Zen winning the first game, again by 1.5 points.

Deep Mind Beat National and Then World Champions 2016-2017

In January 2016 both Facebook and Google DeepMind publish papers about their programs. DeepMind reports that AlphaGo beat the European Champion and Chinese professional Fan Hui by five games to none in an even game match, which took place in October 2015. A first for a computer program.

In March 2016 AlphaGo beats the top 9 dan Korean professional Lee Sedol 4:1 in a five game match in Seoul to world-wide publicity.

January 2017 provides another land-mark, with AlphaGo playing anonymously as Master on several Oriental servers against the very top professionals scoring a resounding 60 wins and no defeats, albeit with short time-limits. So there is no longer any doubt that computers can now play Go better than humans.

March 2017 provided a final match for AlphaGo Master, being retired from competition after beating the acknowledged world’s best player, the Chinese Ke Jie, 3-0.

In October 2017 it was announced that AlphaGo Zero, armed with just the rules, had in 40 days become even better at Go than the original AlphaGo, without the help of game records.

Robotaxi

In October 2018, the California Department of Motor Vehicles issued a permit for Waymo to operate cars without safety drivers. There have been other permits for robotaxi with no safety drivers in the US and China for companies like Waymo, Cruise and Apollo. In 2022, Baidu’s Apollo Go and Pony.ai received permission from Beijing city to remove the safety driver for part of their robotaxi business in a suburb. Cruise has since suspended service. These robotaxi services all used extensive LIDAR (laser radar) and hypermapping for small regions to offer limited services with at most a few hundred vehicles.

Tesla FSD Progress

Tesla has made large progress with its full self driving system. In late 2022, it has gone to end to end neural networks. It is training directly from video. The Tesla FSD 12.X are nearing feature complete and are being assessed as providing human-like driving. Tesla is using far more compute and training data than the equivalent neural net Go program. The shocking completion of the Go programs from 2015 to 2016 is what could be possible. The 2016 equivalent of being competitive with Grandmasters in Go would be a sufficient standard for super high quality Tesla FSD.

Tesla has opened up the service for one month free trials for all Tesla owners in the USA. This usage by 2.5 million drivers will increase the FSD training driving miles from 1 billion to over 2 billion miles. High rates of usage of a quality system will add 0.5 to 2 billion driving miles every month when the system is rolled out in the USA, Canada and China.

17 thoughts on “AI Have Years of Failure then Stunning Success”

  1. Could be that it will be better than us driving in many situations, but I never believe it will be as good as us to a level people will get into the self driving taxis with confidence.
    As long as there won’t be a true AI driving that car with real life expierience of things, how safe will it be / be accepted?

    What would it spair you on the other hand? 50% of a regular cabby vs risk of your life, is that worth it?

    We all know Elon by now, the forecasted prices will be higher, whats the business model actually, is he selling these taxicars to companies or is he doing it himself?

    I still see the level of acceptance years away.

    I myself wont get into one first few years, also what about taxes of these robot taxi drivers?

    Good he’s trying maybe, for some, on the other hand he’s also taking jobs away this way, not really creating, for sure if he want’s tesla to dominate the market himself.

    I said it before, tax them robots.

    • Assuming uber drivers make $10/hr on average and spend half of their working hours driving a fare, so the driver cost while driving is $20/hr.

      A 30 minute Robotaxi ride would save $10 and definitely no need to tip.

      Robotruck is much more compelling – CDL drivers make more. But robotruck is a smaller market because there are fewer trucks than passenger vehicles.

  2. Also, what does “stunning success” look like for driving?

    We focus on safety, but people don’t drive that way. We follow too close, drive too fast, get frustrated when stuck behind someone slower.

    We **could** drive more safely. Many of us choose to accept some risk in exchange for performance.

    Say a robotaxi has 10x better driving skills than me. Is stunning success to use those skills to drive the same as me but with a 90% lower accident rate? A little less aggressive than me and with a 95% lower accident rate? More aggressive than me and with the same accident rate as me?

    I think no matter what FSD does, people will complain about it, because we all complain about other drivers.

    Is “stunning success” possible? If so, what does it look like in real life? Safety is part of the definition, but definitely not 100% of the definition.

  3. It wasn’t clear if it could even now compete with Waymo on limited urban routes for autonomous driving.

  4. One explanation to the lack of objective progress is that fact that Tesla has stuck with HW 3 for years now. And even though – as James Duoma explains – the HW 3 may have plenty of calculation power left to give, the size of the model is limited to the very small RAM of HW 3, i.e. to 16 GB.

    So all the “intelligence” of the FSD SW has to be contained in 16 GB of data. All of it. How to recognize cars, trafic lights, roads, bushes, pedestrians.. How to act in all situations.. That has to be less than 16 GB….

    All of the breakthroughs in LLM’s have come from scaling up the models, but Tesla retains a very small model size of at most 16 GB. If only they could increase the amount of memory by a factor of 10, at least…

  5. Never mind cars. Trains, ships and planes should be much easier than cars (Planes have been mostly self flying for a while, only requiring humans at take off and landing). I wish robo drivers would be introduced on trains in the UK, the buggers went on strike again yesterday. It must be the easiest mode of transport FSD other than lifts (When was the last time you saw a bell boy?).

  6. On the chart, some things are shown by Tesla’s own data to be unsuccessful half or more of the time, including responding to flashing red signals (which actually got worse, from 5 to 2), operating in reverse (the lowest score of 1 forever), understanding special road signs, avoiding potholes, understanding cop hand signals (all the worst: 1). Some things like avoiding debris and stopping for a school bus are simply no longer tested for: NA.
    Things like being bad at reverse will undoubtedly require more cameras to rectify, at added cost and complexity. Current models will never be good at that, and related things like parking (4 currently).
    Most things are short of a “perfect 10” whatever that means in the real world. No human driver is a perfect 10 at anything but if FSD is forever going to be imperfect, the driver can never relax and may even have less response time to “take over” than if he had been driving all along.
    FSD looks like it will hit another plateau soon. And will buyers spend $12k on a feature that never fully works and which might even be arbitrarily dangerous?

    • Yup.

      If I plot the average miles to disengagement, it’s basically flat from end of 2021 to now. The city miles to critical disengagement seemed to make a big jump with 12.3, but now it’s basically back to the level of 11.4.4 in the latest version of FSD (12.3.3). So if you plot it over time, another plateau from end of 2021 to now….

      The only thing that genuinely seems to increase is the usage in terms of miles FSD per month, but that can be partly explained by more FSD users and more cars.

  7. The author fail to understand that levels aren’t linear, they are exponential too. Level 4 is way harder than Level 3, which in turn is way harder than Level 2. Level 5 is basically ASI for driving. Even with the “marvelous” progress in the past weeks, it is still level 2. Regulators are the real benchmark, in every fields, not personal experience, or interviews from Musk or tesla “experts”.

  8. Been on FSD 12.3.3 for the past week in Los Angeles. It definitely requires supervision still.

    Urban driving has improved slightly, but highway driving and mountain driving feels like it’s regressed a bit.

    Still feels like the original repeater cams don’t enough FOV to safely make occluded unprotected lefts. Hopefully Tesla will replace them on older cars.

    • Highway driving is still the old stack. Nothing has to be replaced. Cameras see just fine. Just watch the rendering. It sees all the cars from left and right. The bottleneck is the decisionmaking.

  9. Technological computing power is increasing exponentially. The comparison between advancements in chess decades ago to FSD today are weak arguments at best.

  10. Even if it is 10x safer than me, I’ll still prefer to drive.

    FSD (supervised) seems worth about the same as adaptive cruise control – basically zero.

    Unsupervised is worth big dollars. Supervised is only valuable to make YouTube videos.

    • I agree that real FSD is worth big bucks, but disagree that the current FSD is useless. It would be a godsend for longer (boring) road trips. I would even use it to get to work in the morning, if I had it. But even if I owned a Tesla, ain’t no way in hell I would pay 12k for it.
      If I was rich, sure, but rich is one word that doesn’t describe me.

Comments are closed.