We have updated this article a number of times over the years and this update follows the publication of the Q1 2021 figures.
The August AI day exposed a number of issues that Tesla are working to overcome on the limited beta release that is circulation in a handful of cars. Tesla have also decided to drop the radar from teh Model 3 and Model Y (although the new Model S is quoted as having it) and Tesla have hinted that the Cybertruck will have FSD2 hardware. This all points to a lot of change on a product that Tesla were claiming was nearly ready for full release.
When we look at the numbers, what they show is Tesla have plateau'd in performance and they need a step change in capabilities. The last year has shown no material improvement in the safety of the cars while under autopilot. Passive safety systems are no better, in fact some quarters have resulted in worse figures than the equivalent quarter the year before, and active use of autopilot has seen a 10% increase in accident rates compared to Q1 in 2020.
The numbers presented are a fairly crude "miles per accident". What we don't have is any context behind them, or the numbers of accidents that have actually occurred and registered. The baseline figures are the US NHTSA statistics but is the Tesla data from cars driven in different countries with different safety records? There are also a number of concerns about the basic figures and the comparisons between them which we will cover. We do think it's good that Tesla produce the numbers, we wish they'd acknowledge the blunt comparison they enable is flawed and can lead to incorrect conclusions and even an over reliance on the technology which would result in a relative decrease in safety.
Let us first look at the NHTSAs statistics. While there is some fluctuation, the change year on year is relatively small with little seasonal adjustment.
If we look at the trend of accidents involving Teslas with no AP features including no passive safety we see variability ranging from a low of 102% improvement over the baseline to a high of 442% over the baseline. It's worth noting that this is for cars with no autopilot hardware and no passive safety features, ie these cars have no real technical advantage over any other car. If we take these reaults at face value it appears that as the cars age, the accident rates increase. Our conclusion is either this is meaningless as the sample size is too small, or that both aging cars become more likely to be in an accident, maybe as maintenance deteriorates, and the demographics of Tesla ownership play a significant role in safety improvements being double the average miles between accidents.
The passive safety aspect of autopilot has stayed realtively steadfast at between 3.25x and 3.75x safer than all cars. The removal of radar and the move to Tesla vision will be of interest in this area to see whether the collision avoidance and other warning systems change as a result. What the results do show is there has been no noticeable improvement in the safety systems for a number of years.
Finally, lets look at the active autopilot results. The improvement of active autopilot when compared to the passive use of autopilot has ranged from 135% improvement in Q1 2020 to a 46% improvement in Q4 2019. While the Q1 2021 figures are up against Q4 2020 figures, we have long suspected seasonality as a factor and if we compare like to like quarters, Q1 2021 was actually 10% down on Q1 2020. In terms of absolute numbers, Tesla saw a steady improvement over 2018 until mid 2019, but the last 2 years has seen hardly any improvement.
Our conclusions on this performance is either the launch of the M3 and MY has changed the demographic of owners and an increase in statistically less safe drivers (younger, less affluent) has been matched by genuine improvements in capabilities, or the systems just haven't improved and have plateau'd in their capabilty. This would explain why Tesla feel the need to launch Tesla Vision and do a wholesale change to the design of Autopilot to try and boost performance.
We're pulled the numbers apart, found comparisons that don't look that great, but in reality we suspect there are a host of other factors which we cover below.
If the number of accidents is generally low it renders the results largely meaningless. This may well be the case as Tesla made few cars without even passive safety systems as an example and the number of cars on the road are still relatively not that high. With the average car doing circa 10k miles a year, and in 2020 approx 1m Tesla's on the road, the accident numbers of 2.05M miles between accidents translates to approx 1.2k accidents with cars using passive AP in the quarter. However, if we look at 3.45M on autopilot, we need to factor down the proportion of driving done on AP. There is no published number, but we would suggest the average is maybe 15% of all miles driven. That leads to the number of accidents on autopilot to be around 100 within the quarter. (10k annual mileage = 2.5k quarterly, 15% on AP leads to 375 miles on average across all cars driven on AP, x 1M cars = 375M miles driven). This is becoming statistically insignificant.
One hypothesis could be that in part it is due to the influx of Model 3 owners over the last 2 years, and the demographic of the average Tesla driver has changed to a younger market. It is thought the majority of early Tesla drivers were older and generally professional in occupation as they could afford the higher price point of the car in the market. The Model 3 with its lower price point has opened that up to a wider audience. Its known that drivers between 35 and 60 are the lowest risk group which was almost certainly the primary demographic until the Model 3 launch.
Another potential variable is the buyer attitude. As the buyer type changes from Innovator and early adopter to the early majority there is increased expectation the cars will function fully and be less receptive to the notion of beta software and the warnings that go with it. While the software and release notes warn of the shortcoming, in a world where we have to write 'warning, hot liquid' on the side of coffee cups, the acceptance and dismissal of warning messages is common with the belief these are little more than legal disclaimers and not a cause for real attention.
To back this up, when the demographic of the typical driver is high 30s and above, they are the safest class of drivers per mile with accident rates up to 4x safer than the worst age demographic according to AAAFoundation. As age falls, accident rates go up.
Active autopilot is used predominantly on Motorways/Freeways as this is the road type they were initially limited to and the most common location for use. While it can be used on other road types, we believe the volume of miles will be on this type of road. This road type is however statistically safer than the average road.
UK figures from 2017 state that there were 99 fatalities in 69 Billion miles on motorways v 1068 in 145 Billion rural roads and 626 in 117 Billion urban miles. The data also shows the causality rate is also roughly in proportion to the number of fatalities. While exact numbers may be difficult to determine, the respective ratios of accident rates resulting in injury can be inferred. There can be further complications in the comparison, as an example motorways/freeways may involve higher speed and injury in the event of an accident may be higher than a low speed prang in an urban setting, so there may be significantly more low speed accidents that have gone unreported when comparing relative accident rates.
A reference to the above data can be found in this UK Government report.
Another reference is US Department for transportation road safety which shows interstate roads have approx half the fatality rate compared to all roads) are all relevant factors when making meaningful comparisons.
From these two reports, it's not unreasonable to draw the conclusion that active autopilot should be between 2 and 4x less likely to have an accident purely down to the type of road it is most commonly used on. In Q4 2020 is was only 68% safer.
As well as the road types where AP is engaged, we also believe there are other factors where drivers decide to come off active AP. These include poor weather, complex junctions and heavy traffic where there is a lot of lane changing. All these situations add risk, and as a result, coming off active AP reduces the likelihood of an accident compared to the passive AP side which includes these scenarios.
The system itself does not engage if cameras are obscured with ice or condensation, which are other risk factors.
If drivers genuinely thought AP was always safer then there would be little doubt that it would be used in more situations.
We do not fully subscribe to any argument on the relative safety of Tesla, especially on Autopilot, as the data available is far too weak and any number of assumptions can be made. Through the addition of a small amount of additional data to add context, the picture becomes very different and the argument can be skewed to any political agenda you wish to portray.
It would be beneficial if Tesla put more information out there and allowed some impartial and detailed analysis. They should be mindful that when putting out incomplete data sets that give a superficial view on safety it can do more harm than good and potentially give a false sense of security. We have seen too many instances of people abusing Autopilot, and in a number of occasions with fatal consequences. The suggestion that Autopilot is twice as safe as without must not be taken as an excuse to rely on Autopilot.
But let's take the positives, firstly Tesla are one of the first companies to publish any data like this that we are aware of, that's to be commended. Secondly, the cars appear to be able to cover many more miles between accidents than the average, whether it's the passive safety systems that are also on other similar priced and aged cars or driver demographic we don't know, but if you're in a Tesla, you're safer than average. And thirdly, Tesla like several other car manufacturers, are continuing to innovate and push the bar higher, and while we could be sceptical on the current metrics, merits and safety of the systems, innovation and change is how improvements will come about. For that we should all be grateful. Just don't abuse it.