August 8, 2020: MCU Fail (Again)

It is inevitable. It's out of warranty. Of course.

Back in November of 2015 (about a year after taking delivery) I experienced my first MCU heart attack. One evening I was driving home and decided to reboot the center screen due to glitchy streaming radio. That didn't end so well.

In the last few months within Tesla forums there seems to have been an uptick on discussions about MCU1 failures along with MCU2 upgrade possibilities for older cars. Perhaps Lady Luck would spare me and if my screen ever did give up the ghost, Tesla would be servicing MCU2 upgrades for AP1 cars by then. There have been several iterations of the Model S since its debut in 2012 and mine was built in September 2014 when the very first Autopilot v1 cars rolled off the assembly line. So it's old enough. Maybe by now Tesla is providing MCU2 upgrades for my generation?

But it's 2020. And nothing goes right in 2020.

For the last few weeks the car had apparently been trying to download software update 2020.24.6.11 but never successfully completed.

I figured it was weak Wi-Fi signal to the garage but the mobile app would often experience timeouts as if the car couldn't connect to the cloud at all. The download progress bar in the center display never got past the starting point for weeks.

Eventually impatience got to me and last night I finally took a closer look at the network traffic and discovered that in some cases the car wasn't completing the TCP handshake for certain hosts. For example, the car would perform a DNS query for hermes-sx-prd.vn.tesla.services and then upon name resolution send a TCP SYN to the server IP (hosted on Amazon Web Services). The server would respond with a SYN-ACK as expected, but the car would timeout on the final acknowledgement and never send an ACK to complete the handshake. Being the cutting-edge Silicon Valley shiny shiny that Tesla's OS is, I figured a handy reboot would provide a nice kick against the logjam. Press and hold the two steering wheel scroll knobs...

Aaaaaand it didn't come back on, even after waiting several minutes while the AC kept going (normally the AC should briefly cut out as the MCU begins booting). Crap. It seems I've been 2020ed.

Whenever I reboot something, deep down I worry if the machine will breathe again. It'd be nice if the MCU had a safe mode as a recovery option. I'd even welcome a serial-console terminal interface for that command-line Kung-fu goodness.

So let's quickly recap: car out of warranty (odometer's almost at 121k miles), my second MCU has gone out, the instrument cluster screen has been bubbling for a year (and steadily getting worse), the AC (or maybe it's the battery cooling system) won't shut off immediately when I lock the car, plus pandemic and tons of people are out of work and the United States has achieved greatness in COVID-19 numbers.

There are third-parties such as Gruber Motors who could help, but in that example I'm not prepared to drive all the way to Arizona in the middle of a pandemic. I also wouldn't want to deal with removing the entire MCU and sending it to them (although it'd be a fun effort). My time is rather limited these days for such an endeavor. There are other shops closer to me that are apparently capable of performing these fixes, but I haven't looked into them.

Today I decided to head over to the Fremont Service Center to see if they might possibly have a secret trick up their sleeve (pull fuse 3B in a hidden compartment, boom, fixed?). On the way there, I discovered that the AC still runs (although I can't adjust it). The cooling fans had turned off by the time I walked into the garage this morning which was a good sign, but I had expected it to no longer function while driving. This turns out not to be the case so that was pleasant news. In addition, cruise control and Autosteer actually function. The ultrasonics don't work though, nor are there audible ticks that play along with the blinkers. Autosteer seems to be limited to 45mph. The door handles also don't auto-present when walking up to the car.

The Service Center turns out to be now closed on Saturdays so I scheduled a service appointment through the app. The earliest slot they had was on August 12, a few days away. A follow-up automated text message confirming the appointment came through a moment later.

While inconvenient, the car is still more usable than when my first MCU went out. As I drove home, it became apparent that I've become accustomed to the conveniences of Autosteer over the years. Actively turning the wheel full-time felt cumbersome as if I had returned to the Dark Ages. Such first world problems. I'm no longer in the future, bro. Something as basic as reversing without a rear camera gave me pause. I've become quite reliant on electronics to assist me, so perhaps getting back to the "analog" basics isn't such a bad thing for now.

One caveat - charging does not work (at least at home). After the scheduled start hour the vehicle does not start sipping the outlet. I may have to Supercharge in the next day or two.

Based on what I've been reading in some Tesla forum threads today, it seems like Tesla might finally be offering MCU2 upgrades to old AP1/MCU1 owners. It sounds like it would entail an upgrade on the instrument cluster as well for compatibility. The conspiracy theorist in me however wonders if this MCU1 failure was planned obsolescence since the beginning so they can profit on upgrades. Seems like these failures are all happening right around the same time. Hmm... Or maybe that was the result of all the software tweaks and heavy internal logging left enabled over the years.

But this TMC forum thread raises hope and implies possible reduced upgrade pricing ($1900 vs. $2500), even though parts availability is another question entirely. In the meantime, at least I can still drive. TeslaTap also has an excellent write-up that goes in-depth about this issue and is worth reading.

Update 08/09/2020: it seems that Supercharging does not work either. I also noticed that the car fails to lock itself when I walk away. When I came out of the grocery store earlier the side mirrors had not been folded in. Later back in my garage, I could still push on the door handles and have them auto-present while the key was purposely left in the house. Bummer.