Andrej Karpathy Gives a Technical Update on Tesla Self Driving AI

Andrej Karpathy, Tesla Senior Director of AI gave a technical talk on Tesla’s self-driving AI at the Matroid Scaled Machine Learning Conference 2020.

If Tesla is the first to succeed in the development of full self-driving, then this will enable Tesla to become a multi-trillion company

Tesla has a few dozen developers in the AI team but they have a lot of tools for automation of problem analysis and case harvesting from the data from the car fleet.

Tesla is working on pseudo LIDAR using vision only systems. Tesla is working to achieve primarily neural net-based software and algorithms. They want the neural net systems to learn how to make more accurate depth predictions.

Tesla has a fleet of over million autopilot cars that are uploading and handling many traffic cases. Tesla’s AI team is working through the long tail of exception cases to determine how to drive with complete safety.

There is a lot of complexity. Tesla has tens to hundreds of cases every day of pedestrians moving from between parked cars or being hidden before while in traffic.

There is a lot of complexity with stop signs. Stop signs can be held in someone’s hand and it is inactive until it is raised.

Tesla is curating the test datasets. There are many examples of partially obscured stop signs. They will work to say improve performance on a particular dataset from say 40% to 99%.

They are working towards neural net components that generate a bird-eye view of the vehicle and its surroundings. This is useful for tracking the road edge problems.

18 thoughts on “Andrej Karpathy Gives a Technical Update on Tesla Self Driving AI”

  1. Wait the economy tanks. The public will not be bothered by small number of incidents. And the business will clamor for cheaper ways to do things.

    Reply
  2. My sister’s husband nearly died because he fell, hitting his head, knocking himself unconscious, and the only car available was a manual transmission (930 turbo Porsche) and she couldn’t drive it.

    In her defence, she learned to drive a manual car, then she lost the use of one leg.

    Reply
  3. Unless you have the system turn off if the driver is not paying attention, which is what Tesla’s system does. If you set the turn off time to be short enough, then people will constantly be forced to give the road their attention at certain intervals.

    Reply
  4. Sounds like a scene in a Heinlein(?) novel written in the 1950s. A character is woken alone up in a car by the car beeping to tell her she has to take over driving because her planned route now takes her off the automated driving roads
    Jim Baerg

    Reply
  5. I partially agree. These were the mutterings of JD Power survey ‘participants’. Probably not a group of sophisticated, pro-tech visionaries. But the horse, rotary phone, manual transmission-type failures I think have all contributed to the utter lack of instinct, self-reliance, grit, due diligence, survivability, and just plain common sense lacking in most everyone born after 1985. On the spectrum of ‘earning your tech’, I don’t believe that anyone who buys a car must have the skills to service its basic needs, but being able to use a map/signage, know your basic compass directions, and get out of emergency situations by having an instinct for planning is nearly negligible ‘nowadays’. Yes, there is nothing wrong with not knowing how to take care of an animal and use it for a purpose (horse), not knowing other means of using a phone (landline buttons, rotary, etc), etc. But you look at the quality of the human beings that have spent their life piling up ‘nothing wrongs’ rather than ‘doing rights’ and that has led us to our current ‘sheep’ herd mentality. Inability to write a page, focus on writing of 1000 words or longer, etc., etc. It is not ok to be average and without ambition. It happens. Some are stuck in it – but never should we accept it as a cultural goal. It is not about being a dilettante, just attempting general competence. It leads to your level of enjoyment in life. No skills, dreams, etc., just leads to a swamp of a life. This is why we have so much depression.

    Reply
  6. Damn this comment system. Did they take their inspiration from Clippy?

    The automatic “help” screwed up my formatting and posted before I was ready. I hope you can figure out what I was trying to say. I don’t have to patience to try to edit it (if that is even possitble) or to try to attempt to write the post again and hope to trick Clippy into doing it right this time.

    A pox on your comment system.

    Reply
  7. I see two problems with what you describe, unless I am misunderstanding what you wrote.

    1. A human driver as the safety backup to FSD is a pipe dream. If the car is really driving itself, the human will get bored very quickly and will not be paying attention when the car screws up and a quick intervention is needed. Remember the Uber self driving car that hit a pedestrian in Arizona a couple of years ago? The safety backup driver was watching something on her phone at the critical moment. I don’t believe you can prevent that.
    2. “As long as it’s on balance safer than human drivers it will continue to be accepted.”

    Unless there is full legal cover for the companies building the self-driving cars, some hotshot lawyer will talk a victim or victim’s survivors into filing a lawsuit against the car manufacturer at some point and will win a huge payout. That’s will have the exact opposite effect on public opinion than you say the practice will yield.

    Reply
  8. Concerns about creating a lazy society dependent on technology and with diminished driving skills is a heightened concern

    Most people today don’t know how to ride a horse. That doesn’t seem to be a problem.
    Youth today can’t operate a rotary phone. Also not a problem.
    Numerous other such examples.

    And the connection between not driving and “lazy” is beyond me.

    Reply
  9. Tesla is systematically following the only path that escapes the catch-22 situation. From this point, they put out feature complete FSD and Tesla Network as a ridesharing system, but with owner drivers in place. That allows them to collect data from operating FSD iteratively filling in gaps and also letting everybody see the system in operation.

    There’s no difference between each step in autonomy. Teslas now drive themselves on ramp to off ramp (with humans responsible for taking over if needed). They can add other domains piece by piece the same way. There will be some accidents blamed on the system and a few of them will actually be the system’s fault. As long as it’s on balance safer than human drivers it will continue to be accepted.

    TN gives an opportunity to handle complete trips end to end with FSD AP – for anybody to experience. Eventually it seems unnecessary to insist somebody be sitting in the drivers seat.

    Reply
  10. (continued from above; this limit on post size is a great pain!)

    I agree that self-driving cars won’t be ready this year. I don’t know how long it will take, but I believe self-driving cars won’t ever happen if we don’t start on the non-technical aspects that I’ve mentioned here.

    Reply
  11. (continued from above)

    Perhaps another part is passing very carefully-crafted laws that grant the company that produced the self-driving car immunity from lawsuits once they have met certain criteria, probably including passing tests to evaluate the car’s self-driving capability, and the company carrying a specified level of insurance to compensate victims when the car does get into an accident.

    There probably are other parts that I’m not thinking of.

    I don’t now whether enough of the public can be persuaded that, when the self-driving cars are ready, that the cars should be allowed on the public roads. It might have to be forced on them, which would require some very brave legislators. It would be much better if a campaign was started now to educate the public about the risks, protections, and rewards, and how their interests will be protected when the self-driving cars are allowed on the roads.

    An important point I believe will be true and should be emphasized, is that the abilities of self-driving cars will improve rapidly. Assuming the software can be updated over the air, as Tesla does, when a self-driving car does have an accident, once that accident is analyzed, the correction to the software is made, and the correction is pushed to all the cars, no car will ever make that mistake again. That is a huge advantage over human drivers.

    (continued in next post)

    Reply
  12. If the criteria for allowing self-driving cars on the public roads is that they must never make a mistake, must never do something that results in death of a person, then we never will have self-driving cars.

    The criteria should be that allowing self-driving cars on the public road will result in fewer deaths and less total property damage from accidents.

    I imagine it is impossible to determine when that criteria is met without getting actual experience from allowing self-driving cars to drive on public roads.

    If this isn’t a Catch-22 situation, it is a closely related one. We need some way to escape the Catch-22. I don’t believe I’m smart enough to say how to do that, but I’m pretty confident that THAT is where some energy needs to be put if we are ever going to have self-driving cars on the public roads.

    I believe we won’t get to unrestricted self-driving cars in one giant leap. We will do it in steps. Perhaps part of the solution is for self-driving cars to recognize when the environmental conditions are outside what it is able to handle, and have it safely pull off and stop, waiting for a human to take over.

    (continued in next post)

    Reply
  13. maybe specially designated ‘toll-like’ roads with an electronic gateway that scans your self-drive vehicle, requires acceptance, and then provides a bit of specialized signalling support. start with highways and low-traffic arterials. Geez. I’d pay 5c a mile to get ‘supported’ self-drive for a snooze and bumper-to-bumper traffic helper over a boring 100-mile stretch.

    Reply
  14. other findings:
    “…Uncertainty about timeframe for public availability: Experts anticipate self-driving delivery services will be available in the next four years. However, their predictions for self-driving vehicles available for consumer purchase has jumped out to 18 years; five years later than predicted in the 2019 Q4 study…”
    but:
    “…The National Highway Transportation Safety Administration recently approved Nuro to test its driverless delivery vehicles on public roads in California, which accelerates the feasibility of self-driving delivery coming to market. However, automakers continue to encounter technical hurdles in their quest to achieve reliable self-driving personal vehicles. Coupled with consumer sentiment about the technology, there’s still a very long road ahead….”

    Reply
  15. Sorry folks – i will drink the KoolAid forever, but: JD Power – nearly 10,000 peeps said:
    “…Consumers don’t believe the technology is ready—or that society is ready: Technology failure or error remains the top concern about self-driving technology in both countries, with Canadians being even more worried about it (75% compared with 67% in the US). Canada’s climate and mountainous terrain present a significant challenge as one consumer said of self-driving technology: “Not practical in Canada where there is snow and messy roads. The cameras required cannot see clearly at all times as would be necessary for them to work properly and safely in this climate. I’m basing my statements on the automation in our own car that only works occasionally when the roads are dirty due to camera filth.” American and Canadian consumers also are worried about the law of unintended consequences that will come about as a result of self-driving vehicles. Concerns about creating a lazy society dependent on technology and with diminished driving skills is a heightened concern…”
    Maybe highway driving only – incremental de-activates on off-ramp.

    Reply
  16. To me this update made it painfully clear that Tesla is not going to hit “full self driving” for a least 4 years. Almost certainly more. Painful to me because I paid for it

    Reply

Leave a Comment