While Apple continues (or doesn’t continue) to develop its plans for a self-driving automobile, the company is proposing an advanced high-end technology for an augmented reality (AR) display system that would generate a bunch of information onto a vehicle’s windshield.
In a U.S. Patent Office filing last week, Apple described a graphical overlay on the glass in front of the driver that would provide information about the real-time environment as well as data about objects in the distance — humans included — along with the vehicle’s speed, suspension and other factors.
The system, which may borrow on Apple’s science being employed in its soon-to-be-released Vision Pro headsets, would rely on a variety of sensors to collect information about the surrounding environment. This could include visible light cameras, various types of infrared cameras, as well as ultrasonic and light-based scanning devices for building up a three-dimensional picture of the car's surroundings. The patent also mentions the use of geographical positioning devices and radar devices.
Rather than just overlaying the physical elements of the environment, the AR project would essentially integrate elements into the full-scale projection on a large glass area — a complex operation — in a multi-dimensional model of the outside world.
Another feature buried in the 24-page patent filing under patent point #092 indicated that Apple is working toward a FaceTime application that would sync conversations between occupants in the same vehicle and/or those in another vehicle. Apple notes that "In some embodiments, an augmented reality display system included in a vehicle enables visual communication between an occupant of the vehicle and a remotely located user, including an occupant of a separate vehicle.”
Some of the proposals in the current patent application are not radically new. In 2018, Apple filed for similar system implementations that would flash extensive information onto a “smart” windshield.
Just a year ago at its annual developers event, Apple showed an updated version of CarPlay — it revealed CarPlay transforming an instrument cluster using virtual gauges, adding widgets such as calendar alerts and weather info. Display possibilities in the new CarPlay also provided access to native car info like engine RPM, coolant temperature, HVAC and radio controls.
There’s good reason why Apple CarPlay and Google’s Android Auto are putting the nail in the coffin of automakers’ own smartphone platforms: They work. Prior to the platforms’ rollouts, automakers struggled to marry their in-car touchscreens and knob-and-display infotainment systems with smartphones. Such media player interfaces were and continue to be confusing, frustrating or glitchy.
Enter CarPlay and Android Auto, which use software to largely mirror what’s on a driver’s device – in a streamlined and relatively safe auto-centric way – on a car’s infotainment screen. They include the apps and interfaces owners of the devices already know and appear the same on every in-dash display – whether in a Bentley or a Buick.
What type of phone do you need for Apple CarPlay and Android Auto?
CarPlay is compatible with the iPhone 5 going forward, without needing a separate app. Android Auto, on the other hand, requires downloading an app for devices with Android 9 or below, while devices with Android 10 and later don’t require a separate app.
What apps are available for Android Auto and Apple CarPlay?
Both platforms include native apps for familiar features such as phoning, messaging, music, maps, audiobooks and podcasts, and leverage their parent company’s voice assistant for hands-free functionality. They also integrate popular third-party apps for music streaming, messaging services, navigation, parking, shopping, sports and even EV charging.
More than 600 vehicle models now offer CarPlay, and more than 500 are now compatible with Android Auto. The platforms are also available in aftermarket head units, and in 2018, Mazda added CarPlay support to vehicles from 2014 model-year vehicle forward by way of a dealer-installed upgrade for vehicles equipped with the Mazda Connect system.
Some automakers were initially, and until recently, reluctant to allow Apple and Google into their dashboards – Toyota in particular objected to owners’ usage data being shared automatically with Apple and Google. At the time of this writing, Tesla is the only major automaker that has shunned the platforms and stuck with a proprietary system.
What's the difference between wired and wireless Apple CarPlay and Android Auto?
In most cases, a device needs to be connected to a car’s smartphone-integration USB port. (This port might be labeled as such in your car or designated by a white outline.) But with the introduction of iOS 9 in September 2015, more than 20 automakers now offer wireless CarPlay, while Android had already cut the cord in March 2015. (A handful of automakers offer wireless CarPlay only.)
Wireless CarPlay and Android Auto can’t use a Bluetooth connection, and instead require connecting to a vehicle via Wi-Fi. No, this doesn’t mean they chew through the data allowance for an in-car Wi-Fi hot spot. Since the platforms are data-intensive, they need this more robust connection to the car’s infotainment system.
One downside of wireless Apple CarPlay and Android Auto is a Wi-Fi connection more quickly drains a device’s battery. Of course, you can always plug the device into a USB charge-only port, and if your car has wireless device charging and wireless CarPlay and Android Auto you get the best of both worlds.
Do Apple CarPlay and Android Auto get updated?
In addition to their simplicity and familiarity, one of the biggest advantages of CarPlay and Android Auto over automakers’ smartphone-integration platforms is they’re routinely refreshed through software updates. While a handful of automakers offer over-the-air software updates for their infotainment systems, for all but Tesla it’s not as easy and seamless and having your smartphone update while you sleep, and getting new and enhanced features.
For example, along with adding wireless capability, the iOS 9 update allowed car companies to develop apps that let car occupants operate vehicle-specific features such as climate controls. Subsequent updates added features such as support for third-party navigation apps like Waze and for food ordering and parking services apps.
With the release of iOS 13 in 2019 came one of the biggest changes to CarPlay: a Dashboard feature with a split-screen option that can simultaneously show maps, media info and calendar items. It also added a Settings app that lets users switch between light and dark modes, adjust album art in the Now Playing screen and enable a Do Not Disturb feature.
Last year CarPlay added the availability to unlock a car and start the engine with an iPhone for certain 2021 models. The feature also allows sharing digital “keys” with family or friends and will even work for up to five hours after an iPhone’s battery is dead.
Android Auto has also been consistently updated. Google initially didn’t allow third parties to integrate mapping apps with Android Auto, reserving the platform for its own Google Maps and Waze. But now third-party mapping apps such as Sygic and TomTom AmiGO are available.
In late 2016 Google added the option to run Android Auto as an app on an Android device. In this case, that means via an embedded Android Automotive OS, which so far only a few vehicles run, notably Polestar EVs.
At the beginning of 2018, Google announced the addition of Google Assistant to Android Auto, while in July 2019 the platform got a UI overhaul. Last summer, Android Auto got a major upgrade that added being able to personalize the launch screen directly from your phone, a A-Z quick scrolling feature for lengthy catalogs of songs and contacts list and integration of EV charging, parking and navigation apps.
What limitations do Apple CarPlay and Android Auto have?
Neither system is a complete replacement for a car’s “native” infotainment system. If you want to listen to the radio, be it terrestrial or SiriusXM, you need the car’s interface. Same if you want to change various vehicle functions. Navigation apps such as Google Maps are also reliant on data to be able to search for a new destination and map a route – if you’re in an area without service and haven’t pre-downloaded map data (most people don’t even know you can do that), you won’t have a navigation system. That’s one reason why carmakers continue to offer native navigation systems, although some, such as new systems by Mercedes, offer augmented reality directions in the central screen or head-up display that you definitely won’t get with Google Maps.
What Apple CarPlay can do, or can’t do, also depends on the car. Some lock out CarPlay’s ability to look through Playlists in the music app while the car is in motion, for example, requiring you to select something using Siri. Others, however, keep everything available to you at all times.
What's the difference between Android Auto and Android Automotive OS?
We've already described Android Auto, but you're going to increasingly see the term Android Automotive, which is not the same thing. Instead of providing just an interface to control your smartphone through the car's infotainment system, Android Automotive is in fact the entire infotainment system. It is effectively an operating system created by Google that car brands can customize with their specific designs or "skins" that can control everything a car's native system normally would. So, think changing a radio station or customizing drive mode settings. Android Auto is still needed to communicate and control smartphone apps, and although CarPlay was not yet compatible at the time of this writing, it will in the future. You can see more about Android Automotive and its early adoption in the 2022 GMC Yukon below.