Tesla safety report and the need for caution Q1 2020 Update

Updated 07-05-2020

We wrote an article when Telsa first released the data on accident rates with autopilot that essentially highlighted that the statistics are not comparing like for like. That article is below and the key points of that article still hold true today.

As more results get published we can however start to see more insight into the data, including trend data, which highlights a number of other points which reinforce the view that these things are not like for like comparisons.

Let us first look at the NHTSAs statistics. Over the last year this has fallen from 498k miles to 479k, the worst quarters effectively being winter, but overall, there is fairly low variability in the numbers. If these figures are correct and not averages over a longer time periods then there seems to be little (under 4%) seasonal effect on accident rates.

The second data point to look at is the no AP features including passive. The data has trended

These figures are showing a high degree of variability of nearly 25%. As the NHTSA baseline shows little variability for seasons we need to look for at other hypotheses. One could be the number of accidents is generally low rendering the results largely meaningless and this may well be the case as Tesla made few cars without even passive safety systems. Assuming they are true, the potential reason for these figures being safer than the all comers average may be a reflection on the typical Tesla owner demographic. There seems to be little other variable as none of the active or passive safety systems are available, and perhaps this is the more meaningful reference point and not the NHTSA. Put another way, the typical Tesla owner may well have this rate of accidents regardless of what they drive.

The next set of numbers are the passive AP figures. These have essentially got worse over the last 3 quarters as follows:

One hypothesis could be that passive safety systems are more susceptible to weather conditions than normal drivers (i.e. passive systems do not take into consideration the weather). Alternatively, it could be that with the influx of Model 3 owners over the last 2 years, the demographic of the average Tesla driver has changed. This was one of the points we raised as a variable between the all miles driven and the Tesla figures with the assumption the majority of Tesla drivers were older and generally professional in occupation. The model 3 with its lower price point has opened that up to a wider audience. Another potential variable is the buyer attitude. As the buyer type changes from Innovator and early adopter to the early majority, the expectation from these new buyers is that the cars will perform safely and reliably and not acknowledge the Beta warnings. This would in turn lead to greater risk taking. Finally, drivers may be increasingly using Active AP resulting in the results being skewed as the mix of road type is becoming higher risk. Put differently, if active AP is used increasingly more on the relatively safe freeways, then these 'safe' miles will be excluded from the passive AP results which will in turn make these figures worse. It's difficult to conclude on the true cause, it may well be a mixture of the above points, however the variability in passive safety results is only marginally better than the difference between the passive safety system and no safety system, albeit in each quarter the passive safety systems have been at least 30% better. Given the cause of the variability cannot be explained, it's hard to conclude the software is actually still performing well as the figures getting apparently worse, although we would be surprised if it was now worse.

The final set of numbers is the Active AP. While Q1 figures show a dramatic improvement over Q4, Q4 was a relatively poor quarter. Comparing the last quarters results with the 2 quarters before shows only modest gains.:

We've already discussed the NHTSA and seasonal variability being small and indeed we might reasonably assume that in poor weather in winter the use of active AP is avoided. If we compare the uplift between passive and active AP we see a marked rise in relative terms, Q1 2020 being 135% better, Q3 2019 being 60% better. This is in part because passive AP figures have fallen markedly between the 2 quarters by 35%. If drivers are increasingly determining which conditions are safest and happy to use Active AP more than this could be a sign of increased confidence in active AP. Another consideration is that while the number of miles is large, the statistically important figure is the number of reported accidents, and just like the no active or passive AP figures, if this is relatively small the confidence in the results is reduced.

Our previous comments including driver demographics and Active AP being used on the safest roads (another reference is US Department for transportation road safety which shows interstate roads have approx half the fatality rate compared to all roads) are all relevant factors when making meaningful comparisons.

A lot of numbers have been discussed so what is our TL;DR? Firstly, the points we previously made about active AP being used on the safest roads, at the safest time, in the safest weather and by the safest drivers are all still true. No new evidence has been provided that help address these variables. The trend that 3 quarters of falling passive safety is a concern. While passive safety undoubtedly works the falling performance with no explanation leaves no room for complacency and we believe is a result of the shifting driver demographic as the cheaper M3 is now available to an younger and less affluent market. Finally, active Autopilot performance fluctuates between quarters and while we appreciate the raw numbers are better, we are not satisfied that this can be attributed to the software over the external conditions when in use.

The TL:DR on our TL;DR is that drivers intuitively use active AP when they themselves recognise that the conditions are very safe which probably explains the numbers, and if active AP was genuinely the safest way to drive, why would we any Tesla owner ever chose to disengage it? We simply don't, so we know active AP is not universally safer.

Our original article

Tesla's Vehicle Safety Report stated some interesting, and we use the word lightly, facts.

This is essentially trying to suggest that the use of AP results in half the rate of accidents per mile driven than Tesla not on Autopilot.

Let's break this down a bit.

Superficially the Tesla figures could therefore look quite worrying. Their software is 2x as safe as not using it, but as it is largely used in the context of roads that are historically 4x safer than other roads, it could be argued they should be 4x as safe on a like for like basis, or expressed differently, given the road type it could be argued that AP is half as safe as expected. The mathematics here are simplified, but we trust it illustrates a key problem in interpreting the Tesla data.

We can add in some additional data points. Tesla's are expensive cars and it's also not unreasonable to assume that the demographic of the typical driver is someone in their 30s and above. These are by far the safest class of drivers per mile with accident rates up to 4x safer than the worst age demographic according to AAAFoundation. This is relevant to the overall stats against the field and not AP engaged v not.

There's also an article on general safety features available on modern cars, some of these features only becoming more common place in recent years such as lane assist and so rare on many older cars. We saw material safety improvements when seatbelts became standard and general reduction in accident rates due to ABS and stability aids, we're now seeing a similar increase from these new features. Forbes top 7 car safety features goes into a little more detail.

Technology improving safety is far from new, the insurance industry for highway safety found a 40% reduction in single vehicle accident rates from the introduction of stability control, technology which some cars still don't have and are part of the baseline statistics. What it this suggests is any modern car with passive safety systems have a marked safety improvement of those that don't, and we don't have the data points for a comparable statistic. Tesla could produce statistics that compare pre AP hardware cars, the Mobileye system and their own systems if they wanted to, although in fairness the mileage rates may be small in the earlier models to be statistically meaningful.

But lets conclude positively

We do not fully subscribe to any argument on the relative safety of Tesla, especially on Autopilot, as the data available is far too weak and any number of assumptions can be made. Through the addition of a small amount of additional data to add context, the picture becomes very different and the argument can be skewed to any political agenda you wish to portray.

It would be beneficial if Tesla put more information out there and allowed some impartial and detailed analysis. They should be mindful that when putting out incomplete data sets that give a superficial view on safety it can do more harm than good and potentially give a false sense of security. We have seen too many instances of people abusing Autopilot, and in a number of occasions with fatal consequences. The suggestion that Autopilot is twice as safe as without must not be taken as an excuse to rely on Autopilot.

But let's take the positives, firstly Tesla are one of the first companies to publish any data like this that we are aware of, that's to be commended. Secondly, the cars appear to be able to cover many more miles between accidents than the average, whether it's the passive safety systems that are also on other similar priced and aged cars or driver demographic we don't know, but if you're in a Tesla, you're safer than average. And thirdly, Tesla like several other car manufacturers, are continuing to innovate and push the bar higher, and while we could be sceptical on the current metrics, merits and safety of the systems, innovation and change is how improvements will come about. For that we should all be grateful. Just don't abuse it.

About tesla-info.com

We are Tesla owners and in general supporters. We don't hold stock in Tesla, nor do we short Tesla stock. Our views are our own, based on data points and research that we conduct ourselves. Our goal is to take an impartial view as far too many organisations are either extremely pro or anti Tesla.

Facebook LinkedIn This site uses cookies to deliver our services and to show you relevant ads. By using our site, you acknowledge that you have read and understand our Privacy and Cookie Policy. Your use of the tesla-info website is subject to these policies and terms. All data is provided on a reasonable endeavours basis but errors and omissions may exist. No data should be relied upon as being accurate and additional checks should be made if the information is material to any purchase or use of the car. We provide product listings on Amazon for which we will receive a small commission if you chose to buy.