We have updated this article a number of times over the years and this update follows the publication of the 2020 figures. Timing is also interesting with the latest beta version of autopilot in limited circulation since late 2020 and with a promised roll out in 2021, the need to both understand what these numbers are really saying, and what they are not, and the confidence you can take from them needs to be understood before too much confidence is placed in the Tesla systems. The 4D modelling Tesla say they are now using may improve matters going for4ward if it delivers on its potential, or may encourage an unfounded over reliance on the system and have the reverse effect.
The numbers presented are fairly crude miles per accident. What we don't have is any context behind them, or the numbers of accidents that have actually occurred and registered. The baseline figures are the US NHTSA statistics but is the Tesla data from cars driven in different countries with different safety records? There are also a number of concerns about the basic figures and the comparisons between them which we will cover. We do think it's good that Tesla produce the numbers, we wish they'd acknowledge the blunt comparison they enable is flawed and can lead to incorrect conclusions and even an over reliance on the technology which would result in a relative decrease in safety.
Let us first look at the NHTSAs statistics. While there is some fluctuation the change year on year is relatively small with little seasonal adjustment.
If we look at the trend of accidents involving Tesla's with no AP features including passive safety we see variability ranging from a low of 262% improvement over the baseline to a high of 442% over the baseline. It's worth noting that this is for cars with no autopilot hardware and no passive safety features. The number of cars are relatively small so the sample size could be too small to be statistically significant, but if we take them at face value it suggests the autopilot hardware and software plays no part in an approximate 3x-4x improvement in safety compared to the all cars baseline. Could this be the basic demographics of the Tesla owner community being significantly safer drivers than the average, newer cars with better brakes are able to avoid avoid accidents or something else?
Because of seasonality we believe quarter to the same quarter in the previous year analysis is better than looking at quarter to quarter. When we do so we see an alarming reduction in safety over the last year. Q4 2020 is 23% worse than Q4 2019 for Tesla's with no autopilot features. The cars without the passive safety will be getting older over time and this fall could be due to more mechanical failures or lower maintenance standards resulting in accidents or a change in demographics. The argument then suggests that safety reduces as cars age, and benchmarking new cars (and the majority of Tesla's are now under 2 years old) against all cars will naturally lead to significant improvements based on age.
If we now look at the passive safety figures we can compare them to Tesla's with no passive safety features. We believe this is a more meaningful comparison as the primary change is the addition of the passive safety. The no passive safety figures are falling due to age and we would therefore expect the passive figures to improve on a comparative basis especially give the significant volume increase of new cars in the fleet.
If we do so, we see them fluctuating over time with an all time high of 61% better in Q4 2020. While this looks good in isolation with the best uplift in relative performance being the last quarter we can also compare to the equivalent quarter in previous years. If we compare Q4 2020 to Q4 2019 we see the performance has fallen by 2% (2.05 v 2.1M), and if we do the same for Q3 2020, it is 10% below Q3 2019 (2.42 v 2.7M). The passive safety systems performance are barely any different to those from 2018 when they started producing the statistics despite a number of enhancements to the hardware and software. This confirms that the older cars without passive safety are getting worse, but the cars with passive safety, the majority Model 3 cars built in the last 2 years, have not improved. One hypothesis is that the demographic of typical Tesla owner has changed with the launch of the Model 3 and any improvement in capability has been offset by a reduction in the safety associated with the average Tesla driver. If this is true, then it logical that the Tesla driver community is having a significant baring on the results and comparisons to wider averages.
Finally, lets look at the active autopilot results. The improvement of active autopilot when compared to the passive use of autopilot has ranged from 135% improvement in Q1 2020 to a 46% improvement in Q4 2019. The trend in 2020 was a slow reduction in improvement from that 135%, to 100%, 90% and finally 68% in Q4 2020. Comparing equivalent quarters and absolute figures, the performance in the last reported quarter (Q4 2020) of 3.45M miles between accidents, when compared to 3.34M miles back in Q3 2018 would suggest very little improvement. We may again find that the shift in driver demographic has counteracted the improvements in the safety, but straight facts are the uplift in improvement over passive is a little over 50%. A further causal factor maybe drivers willing to use Autopilot in situations when it is not suitable but if this is the case, the over reliance of the technology needs to be guarded against to prevent inappropriate use.
We're pulled the numbers apart, found comparisons that don't look that great, but in reality we suspect there are a host of other factors which we cover below.
If the number of accidents is generally low it renders the results largely meaningless. This may well be the case as Tesla made few cars without even passive safety systems as an example and the number of cars on the road are still relatively not that high. With the average car doing circa 10k miles a year, and in 2020 approx 1m Tesla's on the road, the accident numbers of 2.05M miles between accidents translates to approx 1.2k accidents with cars using passive AP in the quarter. However, if we look at 3.45M on autopilot, we need to factor down the proportion of driving done on AP. There is no published number, but we would suggest the average is maybe 15% of all miles driven. That leads to the number of accidents on autopilot to be around 100 within the quarter. (10k annual mileage = 2.5k quarterly, 15% on AP leads to 375 miles on average across all cars driven on AP, x 1M cars = 375M miles driven). This is becoming statistically insignificant.
One hypothesis could be that in part it is due to the influx of Model 3 owners over the last 2 years, and the demographic of the average Tesla driver has changed to a younger market. It is thought the majority of early Tesla drivers were older and generally professional in occupation as they could afford the higher price point of the car in the market. The Model 3 with its lower price point has opened that up to a wider audience. Its known that drivers between 35 and 60 are the lowest risk group which was almost certainly the primary demographic until the Model 3 launch.
Another potential variable is the buyer attitude. As the buyer type changes from Innovator and early adopter to the early majority there is increased expectation the cars will function fully and be less receptive to the notion of beta software and the warnings that go with it. While the software and release notes warn of the shortcoming, in a world where we have to write 'warning, hot liquid' on the side of coffee cups, the acceptance and dismissal of warning messages is common with the belief these are little more than legal disclaimers and not a cause for real attention.
To back this up, when the demographic of the typical driver is high 30s and above, they are the safest class of drivers per mile with accident rates up to 4x safer than the worst age demographic according to AAAFoundation. As age falls, accident rates go up.
Active autopilot is used predominantly on Motorways/Freeways as this is the road type they were initially limited to and the most common location for use. While it can be used on other road types, we believe the volume of miles will be on this type of road. This road type is however statistically safer than the average road.
UK figures from 2017 state that there were 99 fatalities in 69 Billion miles on motorways v 1068 in 145 Billion rural roads and 626 in 117 Billion urban miles. The data also shows the causality rate is also roughly in proportion to the number of fatalities. While exact numbers may be difficult to determine, the respective ratios of accident rates resulting in injury can be inferred. There can be further complications in the comparison, as an example motorways/freeways may involve higher speed and injury in the event of an accident may be higher than a low speed prang in an urban setting, so there may be significantly more low speed accidents that have gone unreported when comparing relative accident rates.
A reference to the above data can be found in this UK Government report.
Another reference is US Department for transportation road safety which shows interstate roads have approx half the fatality rate compared to all roads) are all relevant factors when making meaningful comparisons.
From these two reports, it's not unreasonable to draw the conclusion that active autopilot should be between 2 and 4x less likely to have an accident purely down to the type of road it is most commonly used on. In Q4 2020 is was only 68% safer.
As well as the road types where AP is engaged, we also believe there are other factors where drivers decide to come off active AP. These include poor weather, complex junctions and heavy traffic where there is a lot of lane changing. All these situations add risk, and as a result, coming off active AP reduces the likelihood of an accident compared to the passive AP side which includes these scenarios.
The system itself does not engage if cameras are obscured with ice or condensation, which are other risk factors.
If drivers genuinely thought AP was always safer then there would be little doubt that it would be used in more situations.
We do not fully subscribe to any argument on the relative safety of Tesla, especially on Autopilot, as the data available is far too weak and any number of assumptions can be made. Through the addition of a small amount of additional data to add context, the picture becomes very different and the argument can be skewed to any political agenda you wish to portray.
It would be beneficial if Tesla put more information out there and allowed some impartial and detailed analysis. They should be mindful that when putting out incomplete data sets that give a superficial view on safety it can do more harm than good and potentially give a false sense of security. We have seen too many instances of people abusing Autopilot, and in a number of occasions with fatal consequences. The suggestion that Autopilot is twice as safe as without must not be taken as an excuse to rely on Autopilot.
But let's take the positives, firstly Tesla are one of the first companies to publish any data like this that we are aware of, that's to be commended. Secondly, the cars appear to be able to cover many more miles between accidents than the average, whether it's the passive safety systems that are also on other similar priced and aged cars or driver demographic we don't know, but if you're in a Tesla, you're safer than average. And thirdly, Tesla like several other car manufacturers, are continuing to innovate and push the bar higher, and while we could be sceptical on the current metrics, merits and safety of the systems, innovation and change is how improvements will come about. For that we should all be grateful. Just don't abuse it.
We are Tesla owners and in general supporters. We don't hold stock in Tesla, nor do we short Tesla stock. Our views are our own, based on data points and research that we conduct ourselves. Our goal is to take an impartial view as far too many organisations are either extremely pro or anti Tesla.