June 1, 2020: C2 Channels
It's 3:00am and your car is chatting with the mothership while you're asleep...
Raise your hand if this has ever given you pause:
All your data are belong to us.
My garage has spotty Wi-Fi coverage (the car frequently switches between it and LTE) but when it does connect I'm able to capture the Model S' raw IP network traffic and gain insight on what Tesla's operating system does as it phones home to Tesla's command-and-control center. Normally I'd run wire captures via either a switch mirror port or an inline tap and analyze packets with standard libpcap tools like tcpdump, Wireshark, tshark, etc.
But I recently started using Zeek (the open source tool previously known as Bro that's been around for over two decades) through Corelight's NSM@Home project using a Raspberry Pi 4, having it parse traffic streams and then feeding results into a Splunk HTTP Event Collector. Quick background: Zeek is a passive network analysis tool typically used as part of a larger intrusion detection toolset to characterize traffic and help detect/alert/triage suspicious activity. It often complements signature-based systems like Snort or Suricata in larger business IT environments. The commercial variant is supported by Corelight but being able to run a free mini version on a tiny Raspberry Pi and enumerate network activity is a fun way to learn more about the car since the Internet is a natural part of its ecosystem.
My private network is segmented into many different broadcast domains and each IoT device type is given its own IP space, SSID, firewall policy, etc. so there's no cross-contamination or leakage between the car and anything else in my house. When the car is connected to my wireless network, nothing other than the gateway sees it in the IP broadcast space. My personal devices don't share the car's network.
As Zeek watches aggregated traffic via my switch's mirror port output, it auto-analyzes the activity and sends discovery metadata into Splunk (split into Splunk sourcetypes corresponding to Zeek's native log categories). Below are some of the more interesting ones I've seen after a few days. I've barely spent time with Zeek/Bro like I've wanted to in the past, but at least now I have an independent working setup that's not part of a larger package like Security Onion.
It's time to explore...
First, a quick traffic graph using the corelight_conn sourcetype (relates to Zeek's conn.log). You can see how there is occasionally no data for several short time periods. This is likely Wi-Fi not making it into the garage and the car switching to LTE for its Internet phone-home. Note: the Y axis is set to logarithmic scaling here.
DNS requests are always relevant when doing analysis, and since the car is located in its own IP subnet this is relatively easy to figure out rather than pick apart a crowded IP space and all the overlapping chatter.
As you'd expect, most often these are Tesla-specific domain names such as vehicle-files.teslamotors.com, api-prd.vn.tesla.services, and telemetry-prd.vn.tesla.services. It seems the music streaming functions connect to cdn-radiotime-logos.tunein.com, ice5.somafm.com, opml.radiotime.com, server1.chilltrax.com, among others. There are unexpected domains like www.googleapis.com, www.adobe.com, and www.aol.com (maybe I temporarily opened the MCU browser and it triggered a bunch of background lookups?). I generally don't use the in-car browser on MCU1 hardware because 1) it sucks, 2) it's really slow, and 3) it just sucks. Remember when Tesla used to advertise/show-off the fact that you can surf the Internet with it? Yeah, that didn't age well.
We can also see that while a good majority of traffic initiated by the Model S is over TCP 443 (the bits above layer 4 are characterized as "ssl" by Zeek, although this is almost certainly some TLS variant), surprisingly a good portion is also TCP 80/HTTP and Zeek enumerates some HTTP nuances as seen next.
NTP communication is a bit interesting since there doesn't seem to be a corresponding DNS query. For example, one of the IPs is 173.255.215.209 whose PTR record is darwin.kenyonralph.com. The other (44.190.6.254) has no PTR record at all although whois records points to ownership by Amateur Radio Digital Communications. Maybe these are hardcoded in the software? At least 173.255.215.209 is GeoIPed as Fremont, CA so maybe the Tesla software engineers decided to just choose an NTP server closest to the building they were working in.
Curiosity leads us to the corelight_http sourcetype. There are a handful of User-Agents in the midst, mostly curl/7.66.0 (which was released in September last year so the software deployed seems relatively recent). But cid-updater/c6ea01d2f2f39d61 is interesting; I can only assume this is the Center Information Display ("MCU"). tvoice I'm guessing is related to the software for the voice command system (I should check what the PNG file is about). Mozilla/5.0 is used for TuneIn radio, and GStreamer is apparently used for TuneIn streaming functions as well with the User-Agent GStreamer souphttpsrc 1.14.4 libsoup/2.62.3.
I have to wonder why Tesla would still use cleartext traffic for some things in this day and age, although music streaming is probably dependent on the specific partner service capabilities.
There's also the corelight_ssl and corelight_x509 sourcetypes which reveal the TLS connections made, their versions, what the server certificates look like, who issued them, their validity periods, RSA key sizes, etc. Any instance of SSL 3.0, TLS 1.0, or TLS 1.1 with weak ciphersuites would invoke a big frown on my part. Fortunately all connections to Tesla domains are handled via TLS 1.2 with non-CBC ciphersuites. Kudos to the Tesla security operations team.
I find it interesting that the cert for mp.tesla.services is issued via Let's Encrypt. Normally commercial services use a public commercial authority like DigiCert. Otherwise most of Tesla's Internet services certificates are of the two-year variety.
It's interesting to see the "telemetry" domain where both the "eng" and "prd" variants use the same certificate (as shown by the common certificate serial number). Assuming this is the Engineering/Development and Production sides of the company, I would think these would be separated into different infrastructures (or at the very least separate infrastructures shouldn't share the same cert and thus the same private key). Certs are also shared across different AWS regions, likely due to operational practicality.
So that wraps up a basic first step in listening in on the conversation with the mothership. This just scratches the surface and it'll start getting more interesting if we mix/match across sourcetypes and pivot from one perspective to another (often based on a common connection unique identifier between log types not shown in the examples above), so perhaps in time I'll dive deeper into how Tesla interacts with these heavy rolling endpoints with large battery packs in the belly.
Fun future prediction: the mothership will become more resilient to DDoS once SpaceX deploys and matures Starlink, perhaps the ultimate Global Server Load Balancing solution:
It's almost as if...
And this is the new Cyberwing Elon will promise after the Cybertruck, shown here at the upcoming 950 kW-capable High-Orbit Superchargers (in development).
Available for pre-order in 2023. On Mars.