September 13, 2020: App Activity
A closer look at Tesla traffic, the Internet kind.
We take our smartphones and apps for granted as we swipe across the glass and on-screen elements animate, but the real activity happening behind the scenes is invisible. Even if nothing moves on-screen, there’s a near-constant stream of communication taking place between the phone’s OS/installed apps and various cloud services they depend on. As long as an Internet connection exists, persistent callbacks to motherships take place. Tesla's app has expanded in its feature set over the last 6 years and I'm sure the background comms have become more complex. This article is a continuation from my previous examination of the in-car OS several months ago and you'll have to excuse the technical IT jargon.
At the time of this writing, this analysis is based on app version 3.10.8-421 on Android 10 on a Google Pixel 3a. The iOS variant likely behaves virtually the same and perhaps I’ll confirm that another day with a test iPhone sitting near me.
We'll dive into the Tesla app and take a quick look at which Internet domains are accessed and compare some other attributes. This provides a high-level view at the number of cloud service endpoints the app depends on as well as the data-in-flight security based on negotiated TLS versions and cipher suites. Since there are no cleartext HTTP connections being made, we can't discern the actual payload contents unless transparent proxying and TLS decryption via man-in-the-middle/on-the-fly certificate generation is performed by the upstream firewall. Although I didn't test this (I have access to specialized equipment to do this but not the licensing), I have to presume that the app does certificate pinning (if Tesla is serious about their security, which they have a decent track record on) in which case this would be futile anyway.
The activity between the phone and Internet was captured as the packets made their way through my home access point, network switch, and the routing gateway (firewall). In my home network these are separate physical devices that allow me to gain visibility to the raw wire traffic and create a PCAP file to analyze after the fact. I performed specific on-screen actions (launching the app, clicking on button X, invoking function A) at 30-second intervals so it was easy to discern based on individual packet timestamp what on-screen actions are correlated in the PCAP. Not exactly forensics, but it gets us in the ballpark.
I should also note that I had to run this test a couple of times with slightly different home network configurations. Because Android (and likely iOS) have started using DNS over TLS (see RFC 7858) for privacy reasons and thus the FQDN being requested is not visible in the capture during name resolution, on one of the test runs I had to specifically block TCP 853 so the phone's OS would fallback to standard DNS over UDP 53. The TLS handshakes usually (but not always) include the target FQDN in the Client Hello (unless encrypted SNI is used, which doesn't seem to be the case with the Tesla app as of now) so it's nice to have some correlation in the event where actual name lookup data isn't available.
A quick sampling above shows the individual client-server connections and their basic transfer characteristics. The domains involved with the most data transferred Bytes by size were clients4.google.com, ownership.tesla.com, errlog.tesla.com, and static-assets.tesla.com. This will become more clear as we progress in the analysis below.
Tapping the wire...
When the app is launched, a TLS 1.3 connection is immediately made to settings.crashlytics.com and TLS 1.2 to app-measurement.com. Crashlytics and AppMeasurement are Google services used by app developers for tracking potential stability problems for their apps as well as capture analytics regarding app usage (I generally block these on my firewall but temporarily allowed these through for this test).
Several TLS 1.2 connections are then made to owner-api.teslamotors.com, apparently located at a datacenter owned by Quality Technology Services around Palo Alto or Santa Clara, California. owner-api.teslamotors.com is ultimately a CNAME for owner-api.vn.tesla.services and resolves to 205.234.31.120 at the time of this writing except for China where it resolves to 211.147.80.248. It's interesting but not surprising that there's a specific arrangement for users behind The Great Firewall of China. Overall I had expected owner-api.teslamotors.com to be part of a Global Server Load Balancing scheme, but apparently not. I used an external service to run the DNS query for multiple geographic locations.
I clicked down the list of the various app functions such as Climate, Controls, Charging, etc. and it looks like the various existing sessions with owner-api.teslamotors.com were used while I turned the AC on and off, vented the sunroof, etc.
But once I clicked on Location, TLS 1.2 connections to streaming.vn.teslamotors.com (a CNAME for streaming.vn.tesla.services) were made. TLS 1.3 connections were also established to clients4.google.com and csi.gstatic.com. The Google-related services aren't surprising given Tesla's reliance on Google Maps technology. As I pinched/zoomed around the map, there were additional connections to clients4.google.com.
Interestingly, streaming.vn.teslamotors.com resolves to the same IP as owner-api.teslamotors.com (205.234.31.120), even in China. Perhaps the "streaming" service is either not easily replicable across the Pacific or the Chinese government isn't concerned about it. Considering that two distinct services share the same IP and are almost certainly a single VIP on a load balancer, Tesla is likely leveraging the Server Name Indication value in the TLS handshake to handle incoming app requests and directing them to appropriate internal service applications.
Next I clicked into the Service menu, a TLS 1.2 connection to owner-api.teslamotors.com is again made, but TLS 1.3 is also set up for onboarding-pre-delivery-prod.teslamotors.com, ownership.tesla.com, and errlog.tesla.com which are fronted via Akamai's content delivery network space. There's also a TLS 1.3 connection to maps.googleapis.com. If I chose to look at the service detail for a particular service appointment and view the related invoice, the app opens the phone's default web browser and creates a TLS 1.3 connection to ownership.tesla.com (in this case the browser which initiated the connection is DuckDuckGo).
When accessing the Loot Box, a TLS 1.2 connection to owner-api.teslamotors.com is made, and then TLS 1.3 connections to static-assets.tesla.com and www.tesla.com (both being handled via Akamai's CDN).
As I was parsing through my Inbox, some messages had a "Learn More" button that triggered the default web browser to launch and redirect to a relevant site (www.tesla.com, my local utilities provider notice page, etc.) and in some cases content hosted on third-party sites such as vimeo.com would preload as part of those pages such as for Tesla support articles. I also saw a couple of oddball domains like location.teslamotors.com and rumcollector.teslamotors.com show up, both of which are hosted on Akamai's CDN. It's interesting to see the old teslamotors.com domain still being used (this was Tesla's original domain before they were finally able to scoop up tesla.com prior to the Model 3 unveiling).
Side note: here's a message from almost a month ago that sort of predicted the firestorms we're currently witnessing in California. It's been looking like Blade Runner out here lately.
While the Tesla app would likely behave very similarly on iOS, I wonder if usage of app-measurement.com would be as pervasive. I also wonder if the negotiated traffic security is the same as on Android since the underlying operating system libraries being relied on are likely different in subtle ways (reliance on different OpenSSL versions, for example). That said, Apple is generally pretty good about staying up to date on these issues. I may revisit this analysis with my iPhone someday.
The following is a quick look at some of the TLS Client Hello and Server Hello packets, the SNI (Server Name Indication) values, and proposed/negotiated cipher suites as the app accesses various Internet endpoints. Tesla has been pretty good about ensuring solid data-in-flight security. While Google endpoints and Tesla services hosted on Akamai's CDN runs over TLS 1.3, the rest is handled via TLS 1.2.
A tabular summary of my quick test:
App Action | Domain(s) | Notes |
App launch |
settings.crashlytics.com app-measurement.com owner-api.teslamotors.com |
Unknown if the Apple iOS version of the Tesla app would also involve the use of Crashlytics and AppMeasurement. May have to verify this with my test iPhone in the future. |
Climate, Controls, Charging | owner-api.teslamotors.com | Several existing connections (created during app launch) to owner-api.teslamotors.com seem to support these functions. |
Location |
streaming.vn.teslamotors.com clients4.google.com csi.gstatic.com |
The bulk of data transmission by payload size seems to be via clients4.google.com. |
Service |
owner-api.teslamotors.com onboarding-pre-delivery-prod.teslamotors.com ownership.tesla.com errlog.tesla.com maps.googleapis.com |
Unknown why maps.googleapis.com is referenced while in the Service menu and not under Location. Perhaps this is a timing issue within the app performing a background operation sometime after the Location function is exited from. |
Loot Box |
owner-api.teslamotors.com static-assets.tesla.com www.tesla.com |
static-assets.tesla.com and www.tesla.com hosted by Akamai CDN. |
Inbox |
www.tesla.com [misc].vimeo.com location.teslamotors.com rumcollector.teslamotors.com |
Third-party domain references in individual messages are dependent on content matter. There may be others not listed here. |
I didn't exhaustively click on every function within the app to trigger network activity, but this provided a good idea of the number of cloud services the app depends on when the car's information is accessed in real-time. As you interact with your app, visualize these connections being made to various cloud services which make the magic happen.