A notice in my LinkedIn feed mentioned that Brilliant Labs has started shipping its new Frame AR glasses. I briefly met with Brilliant CEO Bobak Tavangar at AWE 2023 (right) and got a short demonstration of its “Monocle” prototype. So, I investigated what Brilliant Labs was doing with its new “Frame.”
This started as a very short article, but as I put it together, I thought it would be an interesting example of making design decisions and trade-offs. So it became longer. Looking at the Frames more closely, I found issues that concerned me. I don’t mean to pick on Brillant Labs here. Any hardware device like the Frames is a massive effort, and they talk like they are concerned about their customers; I am only pointing out the complexities of supporting AI with AR for a wide audience.
While looking at how the Frame glasses work, I came across some information related to the Apple Vision Pro’s brightness (in nits), discussed last time in Apple Vision Pro Discussion Video by Karl Guttag and Jason McDowall. In the same way, the Apple Vision Pro’s brightness is being misstated as “5000 nits,” and the Brilliant Labs Frame’s brightness has been misreported as 3,000 nits. In both cases, the nits are the “potential” out of the display and not “to the eye” after the optics.
I’m also repeating the announcement that I will be at SID’s DisplayWeek next week and AWE next month. If you want to meet, please email meet@kgontech.com.
DisplayWeek (next week) and AWE (next month)
I will be at SID DisplayWeek in May and AWE in June. If you want to meet with me at either event, please email meet@kgontech.com. I usually spend most of my time on the exhibition floor where I can see the technology.
AWE has moved to Long Beach, CA, south of LA, from its prior venue in Santa Clara, and it is about one month later than last year. Last year at AWE, I presented Optical Versus Passthrough Mixed Reality, available on YouTube. This presentation was in anticipation of the Apple Vision Pro.
There is an AWE speaker discount code – SPKR24D , which provides a 20% discount, and it can be combined with Early Bird pricing (which ends May 9th, 2024 – Today as I post this). You can register for AWE here.
Brillian Labs Monocle and Frame used the same basic optical architecture, but it is better hidden in the Frame design. I will start with the Monocle, as it is easier to see the elements and the light path. I was a little surprised that both designs use a very simplistic, non-polarized 50/50 beam splitter with its drawbacks.
Below (left) is a picture of the Monocle with the light path (in green). The Monocle (and Frame) both use a non-polarizing 50/50 beamsplitter. The splitter projects 50% of the display’s light forward and 50% downward to the (mostly) spherical mirror, magnifying the image and moving the apparent focus. After reflecting from the mirror, the light is split again in half, and ~25% of the light goes to the eye. The front project image will be mirrored, with an unmagnified view of the display that will be fairly bright. Front projection or “eye glow” is generally considered undesirable in social situations and is something most companies try to reduce/eliminate in their optical designs.
The middle picture above shows a picture I took of the Monocle from the outside, and you can see the light from the beam splitter projecting forward. Figures 5A and 6 (above right) from Brilliant Labs’ patent application illustrate the construction of the optics. The Monocle is made with two solid optical parts, with the bottom part forming part of the beam splitter and the bottom surface being shaped to form the curved mirror and then mirror coated. An issue with the 2-piece Monocle construction is that the beam splitter and mirror are below eye level, which requires the user to look down to see the image or position the whole device higher, which results in the user looking through the mirror.
The Frame optics work identically in function, but the size and spacing differ. The optics are formed with three parts, which enables Brilliant to position the beam splitter and mirror nearer the center of the user’s line of sight. But as Brilliant Lab’s documentation shows (right), the new Frame glasses still have the virtual (apparent) image below the line of sight.
Having the image below the line of sight reduces the distortion/artifacts of the real world by looking through the beam splitter when looking forward, but it does not eliminate all issues. The top seam of the beam splitter will likely be visible as an out-of-focus line.
The image below shows part of the construction process from a Brilliant Labs YouTube video. Note that the two parts that form the beamsplitter with its 50/50 semi-mirror coating have already been assembled to form the “Top.”
As discussed in Nreal Teardown: Part 1, Clones and Birdbath Basics and its Appendix: Second Type of Birdbath, there are two types of “birdbaths” used in AR. The Birdbath comprises a curved mirror (or semi-mirror) and a beamsplitter. It is called a “birdbath” because the light reflects out of the mirror. The beamsplitter can be polarized or unpolarized (more on this later). Birdbath elements are often buried in the design, such as the Lumus optical design (below left) with its curved mirror and beam splitter.
From 2023 AR/VR/MR Lumus Paper – A “birdbath” is one element of the optics
Many AR glasses today use the birdbath to change the focus and act as the combiner. The most common of these designs is where the user looks through a 50/50 birdbath mirror to see the real world (see Nreal/Xreal example below right). In this design, a polarised beam splitter is usually used with a quarter waveplate to “switch” the polarization after the reflection from the curved semi-mirror to cause the light to go through the beam splitter on its second pass (see Nreal Teardown: Part 1, Clones and Birdbath Basics for a more detailed explanation). This design is what I refer to as a “Look through the mirror” type of birdbath.
Brilliant Labs uses a “Look through the Beamsplitter” type of birdbath. Google Glass is perhaps the most famous product with this birdbath type (below left). This birdbath type has appeared in Samsung patents that were much discussed in the electronic trade press in 2019 (see my 2019 Samsung AR Design Patent—What’s Inside).
LCOS maker Raontech started showing a look through the beamsplitter reference design in 2018 (below right). The various segments of their optics are labeled below. This design uses a polarizing beam splitter and a quarter waveplate.
If you look at the RaonTech or Google Glass splitter, you should see that the beam splitter is the full height of the optics. However, in the case of the Frames and Monocle designs (right), the top and bottom beam splitter seams, the 50/50 mirror coating, and the curved mirror are in the middle of the optics and will be visible as out-of-focus blurs to the user.
Pros and Cons of Look-Through-Mirror versus Look-Through-Beamsplitter
The look-through-mirrorbirdbaths typically use a thin flat/plate beam splitter, and the curved semi-mirror is also thin and “encased in air.” This results in them being relatively light and inexpensive. They also don’t have to deal with the “birefringence” (polarization changing) issues associated with thick optical materials (particularly plastic). The big disadvantage of the look-through-mirror approach is that to see the real world, the user must look through both the beamsplitter and the 50/50 mirror; thus, the real world is dimmed by at least 75%.
The look-through-beamsplitter designs encase the entire design in either glass or plastic, with multiple glued-together surfaces coated or coated with films. The need to encase the design in a solid means the designs tend to be thicker and more expensive. Worse yet, typical injected mold plastics are birefringent and can’t be used with polarized optics (beamsplitters and quartwaveplates). Either heavy glass or higher-cost resin-molded plastics must be used with polarized elements. Supporting a wider FOV becomes increasingly difficult as a linear change in FOV results in a cubic increase in the volume of material (either plastic or glass) and,thus, the weight. Bigger optics are also more expensive to make. There are also optical problems when looking through very thick solid optics. You can see in the Raontech design above how thick the optics get to support a ~50-degree FOV. This approach “only” requires the user to look through the beam splitter, and thus the view of the real world is dimmed by 50% (or twice as much light gets through as the look-through-mirror method).
Pros and Cons Polarized Beam Splitter Birdbaths
Most companies with look-through-mirror and look-through-beamsplitter designs, but not Brilliant Labs, have gone with polarizing beam splitters and then use quarter waveplates to “switch” the polarization when the light reflects off the mirror. Either method requires the display’s light to make a reflective and transmissive pass via the beam splitter. With a non-polarized 50/50 beam splitter, this means multiplicative 50% losses or only 25% of the light getting through. With a polarized beam splitter, once the light is polarized with a 50% loss, with proper use of quarter waveplates, there are no more significant losses with the polarized beamsplitter.
Another advantage of the polarized optics approach is that front-projection can be mostly eliminated (there will be only a little due to scatter). The look-through-mirror method can be accomplished (as discussed in Nreal Teardown: Part 1, Clones and Birdbath Basics) with a second-quarter waveplate and a front polarizer. With the look-through-beamsplitter method, a polarizer before the beamsplitter will block the light that would project forward off the polarized beamsplitter.
As mentioned earlier, using polarized optics becomes much more difficult with the thicker solid optics associated with the look-through-beamsplitter method.
Brilliant Labs Frame Design Decision Options
It seems that at every turn in the decision process for the Frame and Monocle optics, Brilliant Labs chose the simplest and most economical design possible. By not using polarized optics, they gave up brightness and will have significant front projection. Still, they can use much less expensive injection-molded plastic optics that do not require polarizers and quart waveplates. They avoided using more expensive waveguides, which would be thinner but require LCOS or MicroLED (inorganic LED) projection engines, which may be heavier and larger. Although, the latest LCOS and MicroLED engines are getting to be pretty small and light, particularly for a >30-degree FOV (see DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8)).
Frames Brightness to the Eye – Likely >25% of 3,000 nits – Same Problem as Apple Vision Pro Reporting
As discussed in the last article on the Apple Vision Pro (AVP) in the Appendix: Rumor Mill’s 5,000 Nits Apple Vision Pro, reporters/authors constantly make erroneous comparisons of “display-out nits” with one device and to the nits-to-the-eye of other devices. Also, as stated last time, the companies appear to want this confusion by avoiding specifying the nits to the eye as they benefit from reporters and others using display device values.
I could not find an official Brilliant Labs value anywhere, but it seems to have told reporters that “the display is 3,000 nits,” which may not be a lie, but it is misleading. Most articles will dutifully give the “display number” but fail to say that they are “display device nits” and not what the user will see and leave it to the readers to make the mistake, while other reporters will make the error themselves.
The display on Frame is monocular, meaning the text and graphics are displayed over the right eye only. It’s fairly bright (3,000 nits), though, so readability should be good even outdoors in sunlit areas.
As with the Brilliant Labs Monocle – the clip-on, open-source device that came before Frame – information is displayed in just one eye, with overlays being pumped out at around 3,000 nits brightness.
Android Central in androidcentral’s These AI glasses are being backed by the Pokemon Go CEO, who was at least making it clear that it was the display device numbers, but I still think most readers wouldn’t know what to do with this number. They added the tidbit that the panels were made by Sony, and they discussed pulse with modulation (also known as duty cycle). Interestingly, they talk about a short on-time duty cycle causing problems for people sensitive to flicker. In contrast, VR game fans favor a very short on-time duty cycle, what Brad Lynch of SadlyItsBradly refers to as low-persistence) to reduce blurring.
A 0.23-inch Sony MicroOLED display can be found inside one of the lenses, emitting 3,000 nits of brightness. Brilliant Labs tells me it doesn’t use PWM dimming on the display, either, meaning PWM-sensitive folks should have no trouble using it.
Below is a summary of Sony OLED Microdisplays aimed at the AR and VR market. On it, the 0.23 type device is listed with a max lumence of 3,000 nits. However, from the earlier analysis, we know that at most 25% of the light can get through Brilliant Labs Frame birdbath optics or at most 750 nits (likely less due to other optical losses). This number assumes that the device is driven at full brightness and that Brilliant Labs is not buying derated devices at a lower price.
I can’t blame Brilliant Labs because almost every company does the same in terms of hiding the ball on to-the-eye brightness. Only companies with comparatively high nits-to-the-eye values (such as Lumus) publish this spec.
Sony Specifications related to the Apple Vision Pro
The Sony specifications list a 3.5K by 4 K device. The industry common understanding is that Apple designed a custom backplane for the AVP but then used Sony’s OLED process. Notice the spec of 1,000 cd/m2 (candelas per meter squared = nits) at a 20% duty ratio. While favorable for VR gamers wanting less motion blur, the low on-duty cycle time is also a lifetime issue. The display device probably can’t handle the heat from being driven for a high percentage of the time.
It would be reasonable to assume that Apple is similarly restricted to about a 20% on-duty cycle. As I reported last time in the Apple Vision Pro Discussion Video by Karl Guttag and Jason McDowall, I have measured the on-duty cycle of the AVP to be about 18.4% or close to Sony’s 20% for their own device.
The 5,000 nits cited by MIT Tech Review are the raw displays before the optics, whereas the nits for the MQ2 were those going to the eye. The AVP’s (and all other) pancake optics transmit about 11% (or less) of the light from an OLED in the center. With Pancake optics, there is the polarization of the OLED (>50% loss), a transmissive pass, and a reflective pass through a 50/50 mirror, which starts with at most 12.5% (50% cubed) before considering all the other losses from the optics. Then, there is the on-time-duty cycle of the AVP, which I have measured to be about 18.4%. VR devices want the on-time duty cycle to be low to reduce motion blur with the rapid motion of the head and 3-D game. The MQ3 only has a 10.3% on-time duty cycle (shorter duty cycles are easier with LED-illuminated LCDs). So, while the AVP display devices likely can emit about 5,000 nits, the nits reaching the eye are approximately 5,000 nits x 11% x 18.4% = 100 nits.
View Into the Frame Glasses
I don’t want to say that Brilliant Labs is doing anything wrong or that other companies don’t often do the same. Companies often take pictures and videos of new products using non-functional prototypes because the working versions aren’t ready when shooting or because they look better on camera. Still, I want to point out something I noticed with the pictures of the CEO, Bobak Tavangar (right), that was published in many of the articles in the Frames glasses. I didn’t see the curved mirror and the 50/50 beam splitter.
In a high-resolution version of the picture, I could see the split in the optics (below left) but not the darkened rectangle of the 50/50 mirror. So far, I have found only one picture of someone wearing the Frame glasses from Bobak Tavangar’s post on X. It is of a person wearing what appears to be a functional Frame in a clear prototype body (below right). In the dotted line box, you can see the dark rectangle from the 50/50 mirror and a glint from the bottom curved mirror.
I don’t think Brilliant Labs is trying to hide anything, as I can find several pictures that appear to be functional frames, such as the picture from another Tavangar post on X showing trays full of Frame devices being produced (right) or the Forbes picture (earlier in the Optical section).
What was I hoping to show?
I’m trying to show what the Frame looks like when worn to get an idea of the social impact of wearing the glasses. I was looking for a video of someone wearing them with the Frame turned on, but unfortunately, none have surfaced. From the design analysis above, I know they will project a small but bright image view with a mirror image of the display off of the 50/50 mirror, but I have not found an image showing the working device from the outside looking in.
Exploded View of the Frame Glasses
The figure below is taken from Brilliant Lab’s online manual for the Frame glasses (I edited it to reduce space and inverted the image to make it easier to view). By AR glasses standards, the Frame design is about as simple as possible. The choice of two nose bridge inserts is not shown in the figure below.
There is only one size of glasses, which Brilliant Labs described in their AMA as being between a “medium and large” type frame. They say that the temples are flexible to accommodate many head widths. Because the Frames are monocular, IPD is not the problem it would be with a biocular headset.
AddOptics is making custom prescription lenses for the Frames glasses
Bonding to the Frames will make for a cleaner and more compact solution than the more common insert solution, but it will likely be permanent and thus a problem for people whose prescriptions change. In their YouTube AMA, Brilliant Labs said they are working with AddOptics to increase the range of prescription values and support for astigmatism.
They didn’t say anything about bifocal or progressive lens support, which is even more complicated (and may require post-mold grinding). As the virtual image is below the centerline of vision, it would typically be where bifocal and progressive lenses would be designed for reading distance (near vision). In contrast, most AR and VR glasses aim to put the virtual image at 2 meters, considered “far vision.”
The Frame’s basic specs
Below, I have collected the basic specs on the Frame glasses and added my estimate for the nits to the eye. Also shown below is their somewhat comical charging adapter (“Mister Charger”). None of these specs are out of the ordinary and are generally at the low end for the display and camera.
Monocular 640×400 resolution OLED Microdisplay
~750nits to the eye (based on reports of a 3,000 Sony Micro-OLED display device)
(90% on-time duty cycle using an
20-Degree FOV
Weight ~40 grams
1280×720 camera
Microphone
6 axis IMU
Battery 222mAh (plus 149mAh top-up from charging adapter)
With 80mA typical power consumption when operating 0.580 on standby)
CPU nRF52840 Cortex M4F (Nordic ARM)
Bluetooth 5.3
Everything in AR Today is “AI”
Brilliant Labs is marketing the frames as “AI Glasses.” The “AI” comes from Brilliant Lab’s Noa ChatGPT client application running on a smartphone. Brillant Labs says the hardware is “open source” and can be used by other companies’ applications.
I’m assuming the “AI” primarily runs on the Noa cell phone application, which then connects to the cloud for the heavy-lifting AI. According to their video by Brillant Labs, while on the Monocle, the CPU only controls the display and peripherals, they plan to move some processing onto the Frame’s more capable CPU. Like other “AI” wearables, I expect simple questions will get immediate responses while complex questions will wait on the cloud.
Conclusions
To be fair, designing glasses and wearable AR products for the mass market is difficult. I didn’t intend to pick on Brilliant Lab’s Frames; instead, I am using it as an example.
With a monocular, 20-degree FOV below the center of the person’s view, the Frames are a “data snacking” type AR device. It is going to be competing with products like the Human AI projector (which is a joke — see: Humane AI – Pico Laser Projection – $230M AI Twist on an Old Scam), the Rabbit R1, Meta’s (display-less) Ray Ban Wayfarer, other “AI” audio glasses, and many AR-AI glasses similar to the Frame that are in development.
This blog normally concentrates on display and optics, and on this score, the Frame’s optics are a “minimal effort” to support low cost and weight. As such, they have a lot of problems, including:
Small 20-degree FOV that is set below the eyes and not centered (unless you are lucky with the right IPD)
Due to the way the beam 50/50 splitter cuts through the optics, it will have a visible seam. I don’t think this will be pleasant to look through when the display is off (but I have not tried them yet). You could argue that you only put them on “when you need them,” but that negates most use cases.
The support for vision correction appears to lock the glasses to a single (current) prescription.
Regardless of flexibility, the single-size frame will make the glasses unwearable for many people.
The brightness to the eye of probably less than 750 nits is not bright enough for general outdoor use in daylight. It might be marginal if used combined with clip-on sunglasses or if they are used in the shade.
As a consumer, I hate the charger adapter concept. Why they couldn’t just put a USB-C connector on the glasses is beyond me and a friction point for every user. Users typically have dozens of USB-C power cables today, but your device is dead if you forget or lose the adaptor. Since these are supposed to be prescription glasses, the idea of needing to take them off to charge them is also problematic.
While I can see the future use model for AI prescription glasses, I think a display, even one with a small FOV, will add significant value. I think Brillant Labs’s Frames are for early adopters who will accept many faults and difficulties. At least they are reasonably priced at $349, by today’s standards, and don’t require a subscription for basic services without too many complex AI queries requiring the cloud.
Many media outlets, large and small, both text and video, use this blog as a resource for technical information on mixed reality headsets. Sometimes, they even give credit. In the past two weeks, this blog was prominently cited in YouTube videos by Linus Tech Tips (LTT) and Artur Tech Tales. Less fortunately, Adam Savage’s Tested, hosted by Norman Chen in his Apple Vision Pro Review, used a spreadsheet test pattern from this blog to demonstrate foveated rendering issues.
I will follow up with a discussion of Linus’s Tech Tips video, which deals primarily with human factors. In particular, I want to discuss the “Information Density issue” of virtual versus physical monitors, which the LTT video touched on.
Influencing the Influencers On Apple Vision Pro
Linus Tech Tips (LTT)
In their “Apple Vision Pro—A PC Guy’s Perspective,” Linus Tech Tips showed several pages from this blog that were nice enough to prominently feature the pages they were using and the web addresses (below). Additionally, I enjoyed their somewhat humorous physical “simulation” of the AVP (more on that in a bit). LTT used images (below-left and below-center) from the blog to explain how the optics distort the display and how the processing in the AVP is used in combination with eye tracking to reduce that distortion. LTT also uses images from the blog (below-right) to show how the field of view (FOV) changes based on the distance from the eye to the optics.
The Adam Savages Tested video either missed or was incorrect on several points it made:
It missed the point of the blog article that the foveated rendering has problems with spreadsheets when directly rendered from Excel on the AVP instead of mirrored by a MacBook.
It stated that taking pictures through the optics is impossible, which this blog has been doing for over a month (including in this article).
It said that the AVP’s passthrough 3-D perspective was good with short-range but bad with long-range objects, but Linuses Tech tips (discussed later) find the opposite. The AVP’s accuracy is poor with short-range objects due to the camera placement.
It said there was no “warping” of the real world with video passthrough, which is untrue. The AVP does less warping than the Meta Quest 3 and Quest Pro, but it still warps objects less than 0.6 meters (2 feet) away and toward the center to the upper part of the user’s view. It is impossible to be both perspective-correct and not warp with the AVP’s camera placement with near objects; the AVP seems to trade off being perspective-correct to have less warping than the Meta headsets.
Artur’s Tech Tales – Interview on AVP’s Optical Design
Linus Tech Tips on Apple Vision Pro’s Human Factors
Much of the Linus Tech Tips (LTT) videos deal with human factors and user interface issues. For the rest of this article, I will discuss and expand upon comments made in the LTT video. Linus also commented on the passthrough camera’s “shutter angle,” but I moved my discussion on that subject to the “Appendix” at the end as it was a bit off-topic and needed some explanation.
It makes a mess of your face
At 5:18 in the video, Linus takes the headset off and shows the red marks left by the Apple Vision Pro (left), which I think may have been intentional after Linus complained about issues with the headband earlier. For reference, I have included the marks left by the Apple Vision Pro on my face (below-right). I sometimes joke that I wonder if I wear it long enough, it will make a groove in my skull to help hold up the headset.
An Apple person who is an expert at AVP fitting will probably be able to tell based on the marks on our faces if we have the “wrong” face interface. Linus’s headset makes stronger marks on his cheeks, whereas mine makes the darkest marks on my forehead. As I use inserts, I have a fairly thick (but typical for wearing inserts) 25W face hood with the thinner “W” interface, and AVP’s eye detection often complains that I need to get my eyes closer to the lenses. So, I end up cranking the solo band almost to the point where I feel my pulse on my forehead like a blood pressure measuring cuff (perhaps a health “feature” in the future?).
Need for game controllers
For virtual reality, Linus is happy with the resolution and placement of virtual objects in the real world. But he stated, “Unfortunately, the whole thing falls apart when you interact with the game.” Linus then goes into the many problems of not having controllers and relying on hand tracking alone.
I’m not a VR gamer, but I agree with The Verge that AVP’s hand and eye tracking is “magic until it’s not.” I am endlessly frustrated with eye-tracking-based finger selection. Even with the headset cranked hard against my face, the eye tracking is unstable even after recalibration of the IPD and eye tracking many times. I consider eye and hand tracking a good “secondary” selection tool that needs an accurate primary selection tool. I have an Apple Magic Pad that “works” with the AVP but does not work in “3-D space.”
Windows PC Gaming Video Mirroring via WiFi has Lag, Low Resolution, and Compression Artifacts
Linus discussed using the Steam App on the AVP to play games. He liked that he could get a large image and lay back, but there is some lag, which could be problematic for some games, particularly competitive ones; the resolution is limited to 1080p, and compression artifacts are noticeable.
From a “pro” design perspective, it is rather poor on Apple’s part that the AVP does not support a direct Thunderbolt link for both data and power, while at the same time, it requires a wired battery. I should note that the $300 developer’s strap supports a lowish 100Mbs ethernet (compared to USB-C/Thunderbolt 0.48 to 40 Gbs) speed data through a USB-C connector while still requiring the battery pack for power. There are many unused pins on the developer’s strap, and there are indications in the AVP’s software that the strap might support higher-speed connections (and maybe access to peripherals) in the future.
Warping effect of passthrough
In terms of video passthrough, at 13:43 in the video, Linus comments about the warping effect of close objects and depth perception being “a bit off.” He also discussed that you are looking at the world through phone-type cameras. When you move your head, the passthrough looks duller, with a significant blur (“Jello”).
The same Linus Tech Tip video also included humorous simulations of the AVP environment with people carrying large-screen monitors. At one point (shown below), they show a person wearing a respirator mask (to “simulate” the headset) surrounded by three very large monitors/TVs. They show how the user has to move their head around to see everything. LTT doesn’t mention that those monitors’ angular resolution is fairly low, which is why those monitors need to be so big.
Sharing documents is a pain.
Linus discussed the AVP’s difficulty sharing documents with others in the same room. Part of this is because the MacBook’s display goes blank when mirroring onto the AVP. Linus discussed how he had to use a “bizarre workaround” of setting up a video conference to share a document with people in the same room.
Information Density – The AVP Delivers Effectively Multiple Large but Very Low-Resolution Monitors
The peak display resolution in the center of the optics is only 44.4 pixels per degree (human vision it typically better than 60 ppd).
The 2-D/Monitor image must be resampled into 3-D space with an effective resolution loss greater than 2x.
If the monitor is to be viewable, it must be inscribed inside the oval sweet spot of the optics. In the case of the AVP, this cuts off about half the pixels.
While the AVP’s approximate horizontal FOV is about 100 degrees, the optical resolution drops considerably in the outer third of the optics. Only about the center 40-50 degrees of the FOV is usable for high-resolution content.
Simply put, the AVP needs more than double the PPD and better optics to provide typical modern computer monitors’ information/resolution density. Even then, it would be somewhat lacking in some aspects.
Below, show the close-up center (best case) through the AVP’s optics on the (left) and the same image at about the same FOV on a computer monitor (right). Things must be blown up about 2x (linearly) to be as legible on the AVP as on a good computer monitor.
Some current issues with monitor simulation are “temporary software issues” that can be improved, but that is not true with the information density problem.
Linus states in the video (at 17:48) that setting up the AVP is a “bit of a chore,” but it should be understood most of the “chore” is due to current software limitations that could be fixed with better software. The most obvious problems, as identified by Linus, are that the AVP does not currently support multiple screens from a MacBook, and it does not save the virtual screen location of the MacBook. I think most people expect Apple to fix these problems at some point in the near future.
At 18:20, Linus showed the real multiple-monitor workspace of someone doing video editing (see below). While a bit extreme for some people with two vertically stacked 4K monitors in landscape orientation monitors and a third 4K monitor in portrait mode, it is not that far off what I have been using for over a decade with two large side-by-side monitors (today I have a 34″ 22:9 1440p “center monitor” and a 28″ 4K side monitor both in landscape mode).
I want to note a comment made by Linus (with my bold emphasis):
“Vision Pro Sounds like having your own personal Colin holding a TV for you and then allowing it to be repositioned and float effortlessly wherever you want. But in practice, I just don’t really often need to do that, and neither do a lot of people. For example, Nicole, here’s a real person doing real work [and] for a fraction of the cost of a Vision Pro, she has multiple 4K displays all within her field of view at once, and this is how much she has to move her head in order to look between them. Wow.
Again, I appreciate this thing for the technological Marvel that it is—a 4K display in a single Square inch. But for optimal text clarity, you need to use most of those pixels, meaning that the virtual monitor needs to be absolutely massive for the Vision Pro to really shine.“
The bold highlights above make the point about information density. A person can see all the information all at once and then, with minimal eye and head movement, see the specific information they want to see at that moment. Making text bigger only “works” for small amounts of content as it makes reading slower with larger head and eye movement and will tend to make the eyes more tired with movement over wider angles.
To drive the point home, the LTT video “simulates” an AVP desktop, assuming multiple monitor support but physically placing three very large monitors side by side with two smaller displays on top. They had the simulated user wear a paint respirator mask to “simulate” the headset (and likely for comic effect). I would like to add that each of those large monitors, even at that size, with the AVP, will have the resolution capability of more like a 1920x1080p monitor or about half linearly and one-fourth in area, the content of a 4K monitor.
Quoting Linus about this part of the video (with my bold emphasis):
It’s more like having a much larger TV that is quite a bit farther away, and that is a good thing in the sense that you’ll be focusing more than a few feet in front of you. But I still found that in spite of this, that it was a big problem for me if I spent more than an hour or so in spatial-computing-land.
Making this productivity problem worse is the fact that, at this time, the Vision Pro doesn’t allow you to save your layouts. So every time you want to get back into it, you’ve got to put it on, authenticate, connect to your MacBook, resize that display, open a safari window, put that over there where you want it, maybe your emails go over here, it’s a lot of friction that our editors, for example, don’t go through every time they want to sit down and get a couple hours of work done before their eyes and face hurt too much to continue.
I would classify Many of the issues Linus gave in the above quote as solvable in software for the AVP. What is not likely solvable in software are headaches, eye strain, and low angular resolution of the AVP relative to a modern computer monitor in a typical setup.
While speaking in the Los Angeles area at the SID LA One Day conference, I stopped in a Bigscreen Beyond to try out their headset for about three hours. I could wear the Bigscreen Beyond for almost three hours, where typically, I get a spitting headache with the AVP after about 40 minutes. I don’t know why, but it is likely a combination of much less pressure on my forehead and something to do with the optics. Whatever it is, there is clearly a big difference to me. It was also much easier to drink from a can (right) with the Bigscreen’s much-reduced headset.
Conclusion
It is gratifying to see the blog’s work reach a wide audience worldwide (about 50% of this blog’s audience is outside the USA). As a result of other media outlets picking up this blog’s work, the readership roughly doubled last month to about 50,000 (Google Analytics “Users”).
I particularly appreciated the Linus Tech Tip example of a real workspace in contrast to their “simulation” of the AVP workspace. It helps illustrate some human factor issues with having a headset simulate a computer monitor, including information density. I keep pounding on the Information Density issue because it seems underappreciated by many of the media reports on the AVP.
Appendix Linus Comments on AVP’s “Weird Camera Shutter Angle”
I moved this discussion to this Appendix because it involves some technical discussion that, while it may be important, may not be of interest to everyone and takes some time to explain. At the same time, I didn’t want to ignore it as it brings up a potential issue with the AVP.
At about 16:30 in the LTT Video, Linus also states that the Apple Vision Pro cameras use “weird shutter angles to compensate for the flickering of lights around you, causing them [the AVP] to crank up the ISO [sensitivity], adding a bunch of noise to the image.”
From Wikipedia – Example of a 180-degree shutter angle
For those that don’t know, “shutter angle” (see also https://www.digitalcameraworld.com/features/cheat-sheet-shutter-angles-vs-shutter-speeds) is a hold-over term from the days of mechanical movie shutters where the shutter was open for a percentage of a 360-degree rotating shutter (right). Still, it is now applied to camera shutters, including “electronic shutters” (many large mirrorless cameras have mechanical and electronic shutter options with different effects). A 180-degree shutter angle means the shutter/camera scanning is open one-half the frame time, say 1/48th of a 1/24th of a second frame time or 1/180th of a 1/90th of a second frame rate. Typically, people talk about how different shutter angles affect the choppiness of motion and motion blur, not brightness or ISO, even though it does affect ISO/Brightness due to the change in exposure time.
I’m not sure why Linus is saying that certain lights are reducing the shutter angle, thus increasing ISO, unless he is saying that the shutter time is being reduced with certain types of light (or simply bright lights) or with certain types of flickering lights the cameras are missing much of the light. If so, it is a roundabout way of discussing the camera issue; as discussed above, the term shutter angle is typically used in the context of motion effects, with brightness/ISO being more of a side issue.
A related temporal issue is the duty cycle of the displays (as opposed to the passthrough cameras), which has a similar “shutter angle” issue. VR users have found that displays with long on-time duty cycles cause perceived blurriness with rapid head movement. Thus, they tend to prefer display technologies with low-duty cycles. However, low display duty cycles typically result in less display brightness. LED backlit LCDs can drive the LEDs harder for shorter periods to help make up for the brightness loss. However, OLED microdisplays commonly have relatively long (sometimes 100%) on-time duty cycles. I have not yet had a chance to check the duty cycle of the AVP, but it is one of the things on my to-do list. In light of Linus’s comments, I will want to set up some experiments to check out the temporal behavior of the AVP’s passthrough camera.
I have taken thousands of pictures through dozens of different headsets, and I noticed that the Apple Vision Pro (AVP) image is a little blurry, so I decided to investigate. Following up on my Apple Vision Pro’s (AVP) Image Quality Issues – First Impressions article, this article will compare the AVP to the Meta Quest 3 by taking the same image at the same size in both headsets, and I got what many will find to be surprising results.
I know all “instant experts” are singing the praises of “the Vision Pro as having such high resolution that there is no screen door effect,” but they don’t seem to understand that the screen door effect is hiding in plain sight, or should I say “blurry sight.” As mentioned last time, the AVP covers its lower-than-human vision angular resolution by making everything bigger and bolder (defaults, even for the small window mode setting, are pretty large).
While I’m causing controversies by showing evidence, I might as well point out that the AVP’s contrast and color uniformity are also slightly lower than the Meta Quest 3 on anything but a nearly black image. This is because the issues with AVP’s pancake optics dominate over AVP’s OLED microdisplay. This should not be a surprise. Many people have reported “glow” coming from the AVP, particularly when watching movies. That “glow” is caused by unwanted reflections in the pancake optics.
If you click on any image in this article, you can access it in full resolution as cropped from a 45-megapixel original image. The source image is on this blog’s Test Pattern Page. As if the usual practice of this blog, I will show my work below. If you disagree, please show your evidence.
Hiding the Screen Door Effect in Plain Sight with Blur
The numbers don’t lie. As I reported last time in Apple Vision Pro’s (AVP) Image Quality Issues – First Impressions, the AVP’s peak center resolution is about 44.4 pixels per degree (PPD), below 80 PPD, what Apple calls “retinal resolution,” and the pixel jaggies and screen door should be visible — if the optics were sharp. So why are so many reporting that the AVP’s resolution must be high since they don’t see the screen door effect? Well, because they are ignoring the issue of the sharpness of the optics.
Two factors affect the effective resolution: the PPD of optics and the optics’ modulation transfer function sharpness and contrast of the optics, commonly measured by the Modulation Transfer Function (MTF — see Appendix on MTF).
People do not see the screen door effect with the AVP because the display is slightly out of focus/blurry. Low pass filtering/blurring is the classic way to reduce aliasing and screen door effects. I noticed that when playing with the AVP’s optics, the optics have to be almost touching the display to be in focus. The AVP’s panel appears to be recessed by about 1 millimeter (roughly judging by my eye) beyond the best focus distance. This is just enough so that the thinner gaps between pixels are out of focus while only making the pixels slightly blurry. There are potentially other explanations for the blur, including the microlenses over the OLED panel or possibly a softening film on top of the panel. Still, the focus seems to be the most likely cause of the blurring.
Full Image Pictures from the center 46 Degrees of the FOV
I’m going to start with high-resolution pictures through the optics. You won’t be able to see any detail without clicking on them to see them at full resolution, but you may discern that the MQ3 feels sharper by looking at the progressively smaller fonts. This is true even in the center of the optics (square “34” below), even before the AVP’s foveate rendering results in a very large blur at the outside of the image (11, 21, 31, 41, 51, and 61). Later, I will show a series of crops to show the central regions next to each other in more detail.
The pictures below were taken by a Canon R5 (45 Megapixel) camera with a 16mm lens at f8. With a combination of window sizing and moving the headset, I created the same size image on the Apple Vision Pro and Meta Quest Pro to give a fair comparison (yes, it took a lot of time). A MacBook Pro M3 Pro was casting the AVP image, and the Meta Quest 3 was running the Immersed application (to get a flat image) mirroring a PC laptop. For reference, I added a picture of a 28″ LCD monitor taken from about 30″ to give approximately the same FOV as the image from a conventional 4K monitor (this monitor could resolve single pixels of four of these 1080p images, although you would have to have very good vision see them distinctly).
Medium Close-Up Comparison
Below are crops from near the center of the AVP image (left), the 28″ monitor (center), and the MQ3 image (right). The red circle on the AVP image over the number 34 is from the eye-tracking pointer being on (also used to help align and focus the camera). The blur of the AVP is more evident in the larger view.
Extreme Close-Up of AVP and MQ3
Cropping even closer to see the details (all the images above are at the same resolution) with the AVP on the top and the MQ3 on the bottom. Some things to note:
Neither the AVP nor MQ3 can resolve the 1-pixel lines, even though a cheap 1080p monitor would show them distinctly.
While the MQ3 has more jaggies and the screen door effect, it is noticeably sharper.
Looking at the space between the circle and the 3-pixel wide lines pointed at by the red arrow, it should be noticed that the AVP has less contrast (is less black) than the MQ3.
Neither the AVP nor MQ3 can resolve the 1-pixel-wide lines correctly, but the 2- and 3-pixel-wide lines, along with all the text, are significantly sharper and have higher contrast than on the AVP. Yes, the effective resolution of the MQ3 is objectively better than the AVP.
Some color moiré can be seen in the MQ3 image, a color artifact due to the camera’s Bayer filter (not seen by the eye) and the relative sharpness of the MQ3 optics. The camera can “see” the MQ3’s LCD color filters through the optics.
Experiment with Slightly Blurring the Meta Quest 3
A natural question is whether the MQ3 should have made their optics slightly out of focus to hide the screen door effect. As a quick experiment, I tried a (Gaussian) blur of the MQ3’s image a little (middle image below) as an experiment. There is room to blur it while still having a higher effective resolution than the AVP. The AVP still has more pixels, and the person/elf’s image looks softer on the slightly blurred MQ3. The lines are testing for high contrast resolution (and optical reflections), and the photograph shows what happens to a lower contrast, more natural image with more pixel detail.
AVP’s Issues with High-Resolution Content
While Apple markets each display as having the same number of pixels as a 4K monitor (but differently shaped and not as wide), the resolution is reduced by multiple factors, including those listed below:
The oval-shaped optics cut about 25-30% of the pixels.
The outer part of the optics has poor resolution (about 1/3rd the pixels per degree of the center) and has poor color.
A rectangular image must be inscribed inside the “good” part of the oval-shaped optics with a margin to support head movement. While the combined display might have a ~100-degree FOV, there is only about a 45- to 50-degree sweet spot.
Any pixels in the source image must be scaled and mapped into the destination pixels. For any high-resolution content, this can cause more than a 2x (linear) loss in resolution and much worse if it aliases. For more on the scaling issues, see my articles on Apple Vision Pro (Part 5A, 5B, & 5C).
As part of #4 above or in a separate process, the image must be corrected for optical distortion and color as a function of eye tracking, causing further image degradation
Scintillation and wiggling of high-resolution content with any head movement.
Blurring by the optics
The net of the above, and as demonstrated by the photographs through the optics shown earlier, the AVP can’t accurately display a detailed 1920×1080 (1080p) image.
AVP Lack “Information Density”
Making everything bigger, including short messages and videos, can work for low-information-density applications. If anything, the AVP demonstrates that very high resolution is less important for movies than people think (watching movies is a notoriously bad way to judge resolution).
As discussed last time, the AVP makes up the less-than-human angular resolution by making everything big to hide the issue. But making the individual elements bigger means less content can be seen simultaneously as the overall image is enlarged. But making things bigger means that the “information density” goes down, with the eyes and head having to move more to see the same amount of content and less overall content can be seen simultaneously. Consider a spreadsheet; fewer rows and columns will be in the sweet spot of a person’s vision, and less of the spreadsheet will be visible without needing to turn your head.
This blog’s article, FOV Obsession, discusses the issue of eye movement and fatigue using information from Thad Starner’s 2019 Photonic’s West AR/VR/MR presentation. The key point is that the eye does not normally want to move more than 10 degrees for an extended period. The graph below left is for a monocular display where the text does not move with the head-turning. Starner points out that a typical newspaper column is only about 6.6 degrees. It is also well known that when reading content more than ~30 degrees wide, even for a short period, people will turn their heads rather than move their eyes. Making text content bigger to make it legible will necessitate more eye and head movement to see/read the same amount of content, likely leading to fatigue (I would like to see a study of this issue).
ANSI-Like Contrast
A standard way to measure contrast is using a black-and-white checkerboard pattern, often called ANSI Contrast. It turns out that with a large checkerboard pattern, the AVP and MQ3 have very similar contrast ratios. For the picture below, I make the checkerboard bigger to fill about 70 degrees horizontally for each device’s FOV. The optical reflections inside the AVP’s optics cancel out the inherent high contrast of the OLED displays inside the AVP.
The AVP Has Worse Color Uniformity than the MQ3
You may be able to tell that the AVP has a slightly pink color in the center white squares. As I move my head around, I see the pink region move with it. Part of the AVP’s processing is used to correct color based on eye tracking. Most of the time, the AVP does an OK job, but it can’t perfectly correct for color issues with the optics, which becomes apparent in large white areas. The issues are most apparent with head and eye movement. Sometimes, by Apple’s admission, the correction can go terribly wrong if it has problems with eye tracking.
Using the same images above and increasing the color saturation in both images by the same amount makes the color issues more apparent. The MQ3 color uniformity only slightly changes in the color of the whites, but the AVP turns pink in the center and cyan on the outside.
The AVP’s “aggressive” optical design has about 1.6x the magnification of the MQ3 and, as discussed last time, has a curved quarter waveplate (QWP). Waveplates modify polarized light and are wavelength (color) and angle of light-dependent. Having repeatedly switched between the AVP and MQ3, the MQ3 has better color uniformity, particularly when taking one off and quickly putting the other on.
Conclusion and Comments
As a complete product (more on this in future articles), the AVP is superior to the Meta Quest Pro, Quest 3, or any other passthrough mixed reality headset. Still, the AVP’s effective resolution is less than the pixel differences would suggest due to the softer/blurrier optics.
While the pixel resolution is better than the Quest Pro and Quest 3, its effective resolution after the optics is worse on high-contrast images. Due to having a somewhat higher PPD, the AVP looks better than the MQP and MQ3 on “natural” lower-contrast content. The AVP image is much worse than a cheap monitor displaying high-resolution, high-contrast content. Effectively, what the AVP supports is multiple low angular resolution monitors.
And before anyone makes me out to be a Meta fanboy, please read my series of articles on the Meta Quest Pro. I’m not saying the MQ3 is better than the AVP. I am saying that the MQ3 is objectively sharper and has better color uniformity. Apple and Meta don’t get different physics, and they make different trade-offs which I am pointing out.
The AVP and any VR/MR headset will fare much better with “movie” and video content with few high-contrast edges; most “natural” content is also low in detail and pixel-to-pixel contrast (and why compression works so well with pictures and movies). I must also caution that we are still in the “wild enthusiasm stage,” where the everyday problems with technology get overlooked.
In the best case, the AVP in the center of the display gives the user a ~20/30 vision view of its direct (non-passthrough) content and worse when using passthrough (20/35 to 20/50). Certainly, some people will find the AVP useful. But it is still a technogeek toy. It will impress people the way 3-D movies did over a decade ago. As a reminder, 3-D TV peaked at 41.45 million units in 2012 before disappearing a few years later.
Making a headset display is like n-dimensional chess; more than 20 major factors must be improved, and improving one typically worsens other factors. These factors include higher resolution, wider FOV, peripheral vision and safety issues, lower power, smaller, less weight, better optics, better cameras, more cameras and sensors, and so on. And people want all these improvements while drastically reducing the cost. I think too much is being made about the cost, as the AVP is about right regarding the cost for a new technology when adjusted for inflation; I’m worried about the other 20 problems that must be fixed to have a mass-market product.
MTF is measured by putting in a series of lines of equal width and spacing and measuring the difference between the white and black as the size and spacing of the lines change. People typically use 50% contrast critical to specify the MTF by convention. But note that contrast is defined as (Imax-Imin)/(Imax+Imin), so to achieve 50% contrast, the black level must be 1/3rd of the white level. The figure (below) shows how the response changes with the line spacing.
The MTF of the optics is reduced by both the sharpness of the optics and any internal reflections that, in turn, reduce contrast.
Today’s article is just some early findings on the Apple Vision Pro (AVP). I’m working on many things related to the AVP, and it will take me a while to prepare all of them for publishing. Among the things I am doing, I am trying to capture “through the optics” pictures of the AVP, and it is unveiling both interesting information on how the AVP works and the second test pattern I tried “broke” the foveated rending of the AVP.
Having done several searches, I have not seen any “through-the-optics” pictures of the AVP yet. They may be out there, but I haven’t found them. So, I thought I would put up a couple of my test pictures to be (hopefully) the first to publish a picture through the AVP’s optics.
Eye Tracking Display Based Rendering, Maybe “Too smart for its own good”
The AVP is sometimes “too smart for its own good,” resulting in bad visual artifacts. In addition to being used for selection, the AVP’s eye-tracking varies the resolution and corrects color issues (aberrations) in the optics by pre-processing the image. This makes it tricky to photograph because the camera lens looks different to the human eye.
Today, I threw together some spreadsheets to check my ability to take pictures through the AVP optics. I started with two large Excel spreadsheets displayed using the AVP’s native Excel App. One spreadsheet used black text on a white background, which looked like the AVP was making the text and lines look “bolder/thicker” than they should look, but it didn’t act that crazy; the AVP seems to be “enhancing” (not always what you want) the spreadsheet’s readability.
But then I tried inverting everything with white text and lines on a black background, and the display started scintillating in a square box that followed the eye tracking. Fortunately, the AVP’s recording captured the effect in the video below.
I want to emphasize that it is not just the camera or the AVP’s video capture that shows the problem with the foveated rendering; I see it with my own eyes. I have provided the spreadsheets below so anyone with an AVP can verify my findings. I have only tested this with the Excel running on the AVP. The effect is most easily seen if you go into “View” in Excel” and make the view smaller with “-” magnifying glass 3 or 4 times to make the text and boxes smaller.
With its eye-tracking-based rendering, the AVP will be tricky to capture through the optics. The tracking behaves differently with different cameras and lenses. When setting up the camera, I can see the AVP changing colors, sometimes resulting in pictures that are colored differently than what my eye sees.
It seems pretty clear that the AVP is using “foveated,” variable resolution rendering even on still subjects like a spreadsheet. This re-rendering is based on the eyes and due to the change in the 3-D space locking (aka, SLAM) that caused the artifacts seen in the White text and lines on the BLACK spreadsheet.
I should point out that if not for the foveation, the whole image would scintillate. Still, the foveated rendering worsens because it creates a visible square at the boundary between the foveated area and the lower-resolution region. The “foveated rendering” makes it worse by changing the text and lines’ resolution and thickness. I would argue that a more graceful degradation would be to have the whole image rendered the same way (it is not a processing limitation to render a spreadsheet), with the whole image scintillating rather than having boundary lines where it does and does not and with the boldness changing at the boundaries as well. The key point is that the AVP’s display, while much better than almost all other VR/MR headsets, is not, as Apple puts it, “retinal resolution” (or beyond what the eye can see).
Anyway, for the record, below are a couple of through-the-optics test pictures. The first was taken with an R5 camera with a 28mm lens and “pixel shifting” to give a 400-megapixel image. Click on the crop of a very small portion of the center of that picture below to see it in full resolution.
Crop of a very small portion of the original image to show the full detail
The second image below was taken with an Olympus D mark III (Micro Four-Thirds camera) with a 17mm lens. It does not have the resolution of the R5, but the AVP’s eye tracking behaves better with this lens. This camera has a 24mp sensor, and then I used its pixel-shifting feature to capture the image at about 80 megapixels. The whole image (click to see at full resolution) is included below.
If you scroll around the full-resolution image, you can make out the pixel grid through most of the image, yet the text becomes blurrier much more quickly. Preliminarily, this seems to suggest foveated rendering. I have not had time to check yet, but I suspect the resolution falloff coincides with the squares in the white-on-black spreadsheet.
Very Small crop from the image above to show the detail
Conclusion
Designers have to be careful when it comes to applying technology. Sometimes, the same smarts that make one thing work will make others behave poorly.
The biggest part of the problem could be a bug in the AVP software or the Excel port. I’m not saying it is the end of the world, even if it is not improved. There is probably a way to “tone down” the foveated rending to reduce this problem, but I don’t think there is any way to eliminate it, given the display resolution. At the same time, the second test I tried caused it to “break/escape.” Since it happens so readily, this problem will likely show up elsewhere. Fundamentally, it comes down to the display not having a resolution as good as human vision.
Today, I picked up my Apple Vision Pro (AVP) at the Apple Store. I won’t bother you with yet another unboxing video. When you pick it up at the store, they give you a nice custom-made shopping bag for the AVP’s box (left). They give you about a 30-minute guided tour with a store-owned demo headset, and when you are all done with the tour, they give you yours in a sealed box.
iFixit asked if I would help identify some of the optics during their AVP “Extreme Unboxing” (it is Apple; we need a better word for “teardown”). I have helped iFixit in the past with their similar efforts on the Magic Leap One and Meta Quest Pro and readily agreed to help in any way that I could.
iFixit’s “Extreme Unboxing”
As per iFixit’s usual habit, they took the unboxing of a new product to the extreme. They published the first of several videos of their extreme unboxing of the AVP today (Feb. 3rd, 2023). You can expect more videos to follow.
Perhaps the most unexpected thing iFixit showed in the first iFixit video is that the Eyesight (front display) has more than a single lenticular lens in front of the Eyesight’s OLED display. There is a second lens-like element and/or a brightness enhancement film (BEF). BEF films a series of triangular refraction elements that act in one direction, similar to a lenticular lens.
That’s it for today. I mostly wanted to let everyone know about the iFixit extreme unboxing. I have a lot of work to do to analyze the Apple Vision Pro.
Part 1 and Part 2 of this series on the Apple Vision Pro (AVP) primarily covered the hardware. Over the next several articles, I plan to discuss the applications Apple (and others) suggest for AVP. I will try to show the issues with human factors and provide data where possible.
I started working in head-mounted displays in 1998, and we bought a Sony Glasstron to study. Sony’s 1998 Glasstron had an 800×600 (SVGA) display, about the same as most laptop computers in that year, and higher resolution than almost everyone’s television in the U.S. (HDTVs first went on sale in 1998). The 1998 Glasstron even had transparent (sort of) LCD and LCD shutters to support see-through operation.
I can’t say I was surprised to see Apple showing the movie watching on airplanes VR app, as I have seen it again and again over the last 25 years. It makes me wonder how well Apple verified the concepts they showed. As Snazzy Lab’s explained, there were no new apps that Apple showed that had not failed before, and it is not clear they failed due to not having better hardware.
Since the technology for watching videos on a headset has been available for decades, there must be reasons why almost no one (Brad Lynch of SadlyItsBradley says he has) uses a headset to watch movies on a plane. I also realize that some VR fans will watch movies on their headsets, but this, like VR, does not mean it will support mass market use.
As will be shown, the total pixel angular (pixels per degree) resolution of the AVP, while not horrible, is not particularly good for watching movies. But then, the resolution has not been what has stopped people from using VR on airplanes; it has been other human factors. So the question becomes, “Has the AVP solved the human factors problems that prevent people from using headsets to watch movies on airplanes?”
Some Relevant Movie Watching Human Factors Information
In 2019 in FOV Obsession, I discussed an excellent Photonics West’s AR/VR/MR Conference presentation by Thad Starner, the Georgia Institute of Technology and a long-time AR advocate and user.
First, the eye only has high resolution in the fovea, which covers only ~2°. The eye goes through a series of movements and fixations known as saccades. What a person “sees” results from the human vision system piecing together a series of “snapshots” at each saccade. The saccadic movement is a function of the activity and the person’s attention. Also, vision is partially, but not completely, blanked when the eye is moving (see: We thought our eyes turned off when moving quickly, but that’s wrong, and Intrasaccadic motion streaks jump-start gaze correction)
Starner shows the results from a 2017 Thesis by Haynes, which included a study on FOV and eye discomfort. Haynes’ thesis states (page 8 of 303 pages and 275 megabytes – click here to download it):
“Thus, eye physiology provides some basic parameters for potential HWD design. A display can be no more than 55° horizontally from the normal line of sight based on oculomotor mechanical limits. However, the effective oculomotor range places a de facto limit at 45°. Further, COMR and saccadic accuracy suggest visually comfortable display locations may be no more than [plus or minus] 10-20° from the primary position of gaze.”
The bottom line is that the human eye will want to stay within about 20° of the center when watching a movie. Generally, if a user wants to see something more than about 30° from the center of their vision, they will turn their head rather than use just their eyes. This is also true when watching a movie or using a large computer monitor for office-type work.
The Optimum Movie Watching FOV is about 30-40 Degrees
It may shock many VR game players that want 120+ degree FOVs, but SMTPE, which sets the recommendations for movie theaters, says the optimal viewing angle for HDTV is only 30°. THX specifies 40 degrees (Wikipedia and many other sources). These same optimum seating location angles apply to normal movie theaters as well.
The front row of a “normal” movie theater is about 60°, which is usually the last row in a theater where people will want to sit. Most people don’t want to sit in the front rows of a theater because of the “head pong” (as Thad Starner called it) required to watch a movie that is ~60° wide.
While 30°-40° may seem small, it comes back to human factors and a feedback loop of the content generated to work well with typical theater setups. A person in the theater will naturally only see what is happening in the center ~30° of the Screen most of the time, except for some head-turning fast action.
The image content generated outside of ~30° helps give an immersive feel but costs money to create and will not be seen in any detail 99.999% of the time. If you take content generated assuming a nominal 30° to 40°viewing angle and enlarge it to fill 90°, it will cause eye and head discomfort for the user to watch it.
AVP’s Pixels Per Degree Are Below “Retinal Resolution”
Another factor is “angular resolution.” The bands in the chart on the right show how far back from a given size TV with a given resolution must sit before you can’t see the pixels. The metric they use for being “beneficial” is 60ppd or more. Also shown on the chart with the dotted white lines are the SMTPE 30° and THX 40° recommendations.
Apple has not given the exact resolution but stated 23 Million (pixels for both eyes). Assuming a square display, this computes to about 3,400 pixels in each direction. The images in the video look to be about a 7:6 aspect ratio which would work out to about ~3680 by ~3150. Also, the optics cut off some of the display’s pixels for each eye, yet often companies count all the display’s pixels.
Apple didn’t specify the field of view (FOV). One big point of confusion on FOV is that VR headsets are typically quoted for both eyes, including the binocular view combing both eyes. The FOV also varies based on the eye relief from person to person (people’s eye insets, foreheads, and other physical features are different). Reports are that the FOV is “similar” to the Meta Quest Pro, which has a binocular FOV of about 106 degrees. The single-eye FOV is about 90°.
Combining the information from various sources, the net result is about 35 to 42 pixels per degree (ppd). Good human 20/20 vision is said to be ~60ppd. Steve Jobs with the iPhone 6 called 300 pixels per inch at reading distance, which works out to ~60ppd), “retinal resolution.” For the record, people with very good eyesight can see 80ppd
Some people wearing the AVP commented that they could make out some screen door effect consistent with about 35-40ppd. The key point is that the AVP is below 60, so jagged line effects will be noticeable.
Using the THX 40° horizontal FOV standard and assuming the AVP is about 90° horizontally (per eye, 110 for both eyes), ~3680 pixels horizontally, and almost no pixels get cropped, this leaves 3680 x (40/90) = ~1635 pixels horizontally. Using the STMPE 30° gives about 3680 x (30/60) = ~1226 pixels wide.
If the AVP is used for watching movies and showing the movie content “optimally,” the image will be lower than full HD (1920×1080) resolution, and since there are ~40ppd, jaggies will be visible.
While the AVP has “more pixels than a 4K TV,” as claimed, they can’t deliver those pixels to an optimally displayed movie’s 40° or 30° horizontal FOV. Using the full FOV would, in effect, put you visually closer than the front row of a movie theater, not where most people would want to watch a movie.
Still, resolution and jaggies alone are not so bad as they would not, and have not, stopped people from using a VR headset for movies.
Vestibulo–Ocular Reflex (VOR) – Stabilizing the View with Head Movement – Simple Head Tracking Fails
The vestibulo-ocular reflex (VOR) stabilizes a person’s gaze during head movement. The inner ear detects the rotation, and if one is gazing, it causes the eyes to rotate to counter the movement to stay fixed on where the person is gazing. In this way, a person can, for example, read a document even if their head is moving. People with a VOR deficiency have problems reading.
Human vision will automatically suppress the VOR when it is a counter product. For example, the VOR reflex will be suppressed if one is tracking an object with a combination of head and eye movement, whereas VOR would be counter-productive. The key point is that the display system must account for the combined head and eye movement to generate the image without causing a vestibular (motion sickness) problem where the inner ear does not agree with the eyes.
Quoting from the WWDC 2023 video at ~1:51:18:
Running in parallel is a brand-new chip called R1. This specialized chip was designed specifically for the challenging task of real-time sensor processing. It processes input from 12 cameras, five sensors, and six microphones.
In other head-worn systems, latency between sensors and displays can contribute to motion discomfort. R1 virtually eliminates lag, streaming new images to the displays within 12 milliseconds. That’s eight times faster than the blink of an eye!
Apple did not say if the “12 cameras” included eye-tracking cameras, as they only showed the cameras on the front, but likely they are included. Complicating matters further is the saccadic movement of the eye. Eye tracking can know where the eye is aimed, but not what is seen. The AVP is known to have superior eye tracking for selecting things from a menu. But we don’t know if the eye tracking coupled with the head tracking deals with VOR, and if so, whether it is accurate and fast enough to solve to not cause VOR-related problems for the user.
Movies on AVP (and VR) – Chose Your Compromises
Now consider some options for displaying a virtual screen on a headset below. Apple has shown locking the Screen in the 3-D space. For their demos, they appear to have gone with a very large (angularly) virtual screen for the demo impact. But, as outlined below, making a very large virtual screen is not the best thing to do for more normal movie and video watching. No matter which option is chosen below, jaggies and “zipper/ripple” antialiasing artifacts will be visible at times due to the angular resolution (pdd) of the AVP.
Simplistic Option: Scale the image to full Screen for the maximum size and have the Screen moves with the headset (not locked in the virtual 3-D space). This option is typically chosen for headsets with smaller FOVs, but it is a poor choice for headsets with large FOVs.
It is like sitting in a movie theater’s front row (or worse).
The screen moves unnaturally with head motion as it follows any head motion.
Lock the Virtual Screen but nearly fill the FOV: This is what I will call “Head-Lock for Demos Only Mode.” If the virtual Screen nearly fills the FOV, then the small head movement will cause the Screen to cut off and will, in turn, will trigger a person’s peripheral vision causing some distraction. To avoid distraction, the user must limit head movement and eye movement; perhaps doable in a short demo, but not a comfortable way to watch a movie.
Locking Screen in 3-D space with the Screen at STMPE 30° to THX 40°: With ~40° FOV, there is room for the head to turn and total without cutting off the Screen or forcing the user to keep their head rigidly held in one location.
This will test the ability of the system to track head motion without causing motion sickness. There will always be some motion-to-photon lag and some measurement errors. There is also the VOR issue discussed earlier and whether it is solvable.
Some additional loss in resolution and potential for motion/temporal artifacts as the flat or 3-D movie is resampled into the virtual space.
Add motion blur to deal with head and eye movement (unlikely as it would be really complex).
The AVP reshows a 24 fps movie four times at 96Hz – does each frame get corrected at 96Hz, and what about visual artifacts when doing so?
What does it do for 30 fps and 60 fps video?
The Screen will still unnaturally be cut off if the user’s head turns too far. It does not “degrade gracefully” as a real-world screen would when you turn away from it.
Apple showed (above) images that might fill about 70 to 90 degrees of the FOV in its short Avatar demos (case 2 above). This will “work” in a demo to be something new and different, but as discussed in #2 above, it is not what you would want to do for a long movie.
And You Are on a Plane and Wearing A Heavy Headset Pressed Against Your Face with a Cord to Snag
On top of all the other issues, the headset processing and sensor must address vestibular-related motion sickness problems caused by being in a moving vehicle while displaying an image.
You then have the ergonomic issues of wearing a somewhat heavy, warm headset sealed against your face with no air circulation for hours while on a plane. Then you have the snag hazard of the cord, which will catch on just about everything.
There will be flight attendants or others tapping you to get your attention. Certainly, you don’t want the see-through mode to come on each time somebody walks by you in the aisle.
A more basic practical problem is that a headset takes up more room/volume due to its shape and the need to protect the glass front than a smartphone, tablet, or even a moderately sized laptop.
Conclusions
It is important to note that humans understand what behaves as “real” versus virtual. The AVP is still cutting off much of a person’s peripheral vision. Something like VOR and Vergence-Accommodation Conflict (VAC discussed in Part 2) and the way focus behaves are well-known issues with VR, but many more subtle issues can cause humans to sense there is something just not right.
In visual human factors, l like to bring up the 90/90 rule, which states, “it takes 90% of the effort to get 90% of the way there, and then the other 90% of the effort to solve the last 10%.” Sometimes this rule has to be applied recursively where multiples of the “90%” effort are required. Apple could do a vastly better job of head and eye tracking with faster response time, and yet people would still prefer to watch movies and videos on a direct-view display.
Certainly, nobody will be the wiser in a short flashy demo. The question is whether it will work for most people watching long movies on an airplane. If it does, it will break a 25+ year losing streak for this application.
I want my readers to know about my first article on Display Daily as a “Senior Analyst” and the article I wrote for the March/April SID Information Display. I will be speaking and attending the upcoming AWE 2023 Conference from May 31st through June 2nd. I also recently recorded another AR Show Podcast with Jason McDowell, which should be published in a few weeks.
I also wanted you to know I have a lot of travel planned for May, so there may not be much, if anything, published on this blog in May. But I have several articles in the works for this month and should have more to discuss in June.
Display Daily Article On Apple – New “Senior Analyst”
Display Daily, a division of Jon Peddie research, has just put out an article by me discussing long-rumored Apple Mixed Reality headset. In some ways, this follows up an article I wrote for Display Daily in 2015 titled VR and AR Head Mounted Displays – Sorry, but there is no Santa Claus.
Display Daily and I are looking at joint wrote and video projects. I will be teaming up with Display Daily as a “Senior Analyst” on these new projects while I continue to publish this blog.
SID Information Display Magazine’s March/April 2023 Article, “The Crucial Role of Optics in AR/MR”
I was asked to contribute an article to SID’s Information Display Magazine’s printed and online March/April 2023 issue.
The article (available for free download) discusses the most common types of optics and displays used in mixed reality today and what I see as the technologies of the future.
Attending and Presenting at AWE 2023
AWE has been the best conference for seeing a wide variety of AR, VR, and MR headsets for many years. While I mostly spend my time on the show floor and in private meetings to see the “good stuff,” I have been invited to give a presentation this year. The Topic of the presentation will be the pros and cons of Optical versus Video Passthrough Mixed Reality. The conference runs from May 31st to June 2nd. I will be presenting at 9:00 AM on June 2nd.
My Long History with Display Daily (20+ Years) and even Longer with Jon Peddie (40+ Years)
I’ve been interacting with Display Daily and its former parent company Insight Media headed by Chris Chinnock, who is still a Display Daily Contributor since I left Texas Instruments to work on LCOS display devices in 1998. Meko, headed by Bob Raikes, took over Display Daily in 2014, then late in 2022, Jon Peddie Research acquired Display Daily.
It turns out that I had known market analyst Jon Peddie since the mid-1980s when he was the chief architect of the TMS34010, the world’s first fully programmable graphics processor, and led the definition of other graphics devices, including the first Video DRAM. Jon suggested we work together on some projects, and I have become a Senior Analyst at Display Daily.
Introduction – Contrast in Approaches and Technologies
This article will compare and contrast the Vuzix Ultralight, Lumus Z-lens, and DigiLens Argo waveguide-based AR prototypes I saw at CES 2023. I discussed these three prototypes with SadlyItsBradly in our CES 2023 video. It will also briefly discuss the related Avegant’s AR/VR/MR 2022 and 2023 presentations about their new smaller LCOS projection engine and Magic Leap 2’s LCOS design to show some other projection engine options.
It will go a bit deeper into some of the human factors of the Digitlens’ Argo. Not to pick on Digilens’ Argo, but because it has more features and demonstrates some common traits and issues of trying to support a rich feature set in a glasses-like form factor.
When I quote various specs below, they are all manufacturer’s claims unless otherwise stated. Some of these claims will be based on where the companies expect the product to be in production. No one has checked the claims’ veracity, and most companies typically round up, sometimes very generously, on brightness (nits) and field of view (FOV) specs.
This is a somewhat long article, and the key topics discussed include:
MicroLED versus LCOS Optical engine sizes
The image quality of MicroLED vs. LCOS and Reflective (Lumus) vs. Diffractive waveguides
The efficiency of Reflective vs. Diffractive waveguides with MicroLEDs
The efficiency of MicroLED vs. LCOS
Glasses form factor (using Digilens Argo as an example)
They are each about 38 grams in weight, including frames, processing, wireless communication, and batteries, and wirelessVuzix developed their own diffractive waveguide and support about a 30-degree FOV. Both are self-contained with wireless, with an integrated battery and processing.
Vuzix developed their own glass diffractive waveguides and optical engines for the Ultralight. They claim a 30-degree FOV with 3,000 nits.
Oppo uses resin plastic waveguides, and MicroLED optical engine developed jointly with Meta Bounds. I have previously seen prototype resin plastic waveguides from other companies for several years. This is the first time I have seen them in a product getting ready for production. The glasses (described in a 1.5-minute YouTube/CNET video) include microphones and speakers for applications, including voice-to-text and phone calls. They also plan on supporting vision correction with lenses built into the frames. Oppo claims the Air Glass 2 has a 27-degree FOV and outputs 1,400 nits.
Lumus Z-Lens
Lumus’s Z-Lens (third from the top right) supports up to a 2K by 2K full/true color LCOS display with a 50-degree FOV. Its FoV is 3 to 4 times the area of the other three headsets, so it must output more than 3 to 4 times the total light. It supports about 4.5x the number of pixels of the DigiLens Argo and over 13x the pixels of the Vuzix Ultralite and Oppo Air Glass 2.
The Z-Lens prototype is a demonstration of display capability and, unlike the other three, is not self-contained and has no battery or processing. A cable provides the display signal and power for each eye. Lumus is an optics waveguide and projector engine company and leaves it to its customers to make full-up products.
Digilens Argo
The DigiLens Argo (bottom, above right) uses a 1280 by 720 full/true color LCOS display. The Argo has many more features than the other devices, with integrated SLAM cameras, GNSS (GPS, etc.), Wi-Fi, Bluetooth, a 48mp (with 4×4 pixel “binning” like the iPhone 14) color camera, voice recognition, batteries, and a more advanced CPU (Qualcomm Snapdragon 2). Digilens intends to sell the Argo for enterprise applications, perhaps with partners, while continuing to sell waveguides optical engines as components for higher-volume applications. As the Argo has a much more complete feature set, I will discuss some of the pros and cons of some of the human factors of the Argo design later in this article.
Through the Lens Images
Below is a composite image from four photographs taken with the same camera (OM-D E-M5 Mark III) and lens (fixed 17mm). The pictures were taken at conferences, handheld, and not perfectly aligned for optimum image quality. The projected display and the room/outdoor lighting have a wide range of brightness between the pictures. None of the pictures have been resized, so the relative FoVs have been maintained, and you get an idea of the image content.
The Lumus Z-lens reflective waveguide has a much bigger FOV, significantly more resolution, and exhibits much better color uniformity with the same or higher brightness (nits). It also appears that reflective waveguides have a significant efficiency advantage with both MicroLEDs (and LCOS), as discussed in MicroLEDs with Waveguides (CES & AR/VR/MR 2023 Pt. 7). It should also be noted that the Lumus Z-lens prototype has only the display with optics and has no integrated processing, communication or battery. In contrast, the others are closer to full products.
A more complex issue is that of power consumption versus brightness. LCOS engines today are much more efficient for an image with full-screen bright images (by 10x or more) than MicroLEDs with similar waveguides. MicroLED’s big power advantage occurs when the content is sparse, as the power consumption is roughly proportional to the average pixel value, whereas, with LCOS, the whole display is illuminated regardless of the content.
If and when MicroLEDs support full color, the efficiency of nits-per-Watt will be significantly lower than monochrome green. Whatever method produces full color will detract from the overall electrical and optical efficiency. Additionally, color balancing for white requires adding blue and red light with lower nits-per-Watt.
Some caveats:
The Lumus Z-Lens is a prototype and does not have all the anti-reflective and other coatings of a production waveguide. Lumus uses an LCOS device with about ~3-micron pixels, which fits 1440 by 1440 within the ~50-degree FOV supported by the optics. Lumus is working with at least one LCOS maker to get an ~2-micron pixel size to support 2K by 2K resolution with the same size display. The image is cut off on the right-hand side of the image by the camera, which was rotated into portrait mode to fit inside the glasses.
The Digilens through the lens image is from Photonics West in 2022 (about one year old). Digilens has continued to improve its waveguide since this picture was taken.
The Vuzix picture was taken via Vuzix Shield, which uses the same waveguide and optics as the Vuzix Ultralight.
The Oppo image was taken at the AR/VR/MR 2023 conference.
Optical Engine Sizes
Vuzix has an impressively small optical engine driving Vuzix’s diffractive waveguides. Seen below left is a comparison of Vuzix’s older full-color DLP engine compared with an in-development color X-Cube engine and the green MicroLED engine used in the Vuzix Ultralite™ and Shield. In the center below is an exploded view of the Oppo and Meta Bound glasses (joint design as they describe it) with their MicroLED engine shown in their short CNET YouTube video. As seen in the still from the Oppo video, they have plans to support vision correction built into the glasses.
Below right is the Digilens LCOS engine, which uses a fairly conventional LCOS (using Ominivision’s LCOS device with driver ASIC showing). The dotted line indicates where the engine blocks off the upper part of the waveguide. This blocked-off area carries over to the Argo design.
The Digilens Argo, with its more “conventional” LCOS engine, requires are large “brow” above the eye to hide it (more on this issue later). All the other companies have designed their engine to avoid this level of intrusion into the front area of the glasses.
Lumus had developed their 1-D pupil-expanding reflective waveguide for nearly two decades, which needed a relatively wide optical engine. With the 2-D Maximus waveguide in 2021 (see: Lumus Maximus 2K x 2K Per Eye, >3000 Nits, 50° FOV with Through-the-Optics Pictures), Lumus demonstrated their ability to shrink the optical engine. This year, Lumus further reduced the size of the optical engine and its intrusion into the front lens area with their new Z-lens design (compare the two right pictures below of Maximus to Z-Lens)
Shown below are frontal views of the four lenses and their optical engines. The Oppo Air Glass 2 “disguises” the engine within the industrial design of a wider frame (and wider waveguide). The Lumus Z-Lens, with a full color about 3.5 times the FOV as the others, has about the same frontal intrusion as the green-only MicroLED engines. The Argo (below right) stands out with the large brow above the eye (the rough location of the optical engine is shown with the red dotted line).
Lumus Removes the Need for Air Gaps with the Z-Lens
Another significant improvement with Lumus’s Z-Lens is that unlike Lumus’s prior waveguides and all diffractive waveguides, it does not require an air gap between the waveguide’s surface and any encapsulating plastics. This could prove to be a big advantage in supporting integrated prescription vision correction or simple protection. Supporting air gaps with waveguides has numerous design, cost, and optical problems.
A typical full-color diffractive waveguide typically has two or three waveguides sandwiched together, with air gaps between them plus an air gap on each side of the sandwich. Everywhere there is an air gap, there is also a desire for antireflective coatings to remove reflections and improve efficiency.
Avegant and Magic Leap Small LCOS Projector Engines
Older LCOS projection engines have historically had size problems. We are seeing new LCOS designs, such as the Lumus Z-lens (above), and designs from Avegant and Magic Leap that are much smaller and no more intrusive into the lens area than the MicroLED engines. My AR/VR/MR 2022 coverage included the article Magic Leap 2 at SPIE AR/VR/MR 2022, which discusses the small LCOS engines from both Magic Leap and Avegant. In our AWE 2022 video with SadlyItsBradley, I discuss the smaller LCOS engines by Avegant, Lumus (Maximus), and Magic Leap.
Below is what Avegant demonstrated at AR/VR/MR 2022 with their small “L” shaped optical engines. These engines have very little intrusion into the front lenses, but they run down the temple of the glasses, which inhibits folding the temple for storage like normal glasses.
At the AR/VR/MR 2023, Avegant showed a newer optical design that reduced the footprint of their optics by 65%, including shortening them to the point that the temples can be folded, similar to conventional glasses (below left). It should be noted that what is called a “waveguide” in the Avegant diagram is very different from the waveguides used to show the image in AR glasses. Avegants waveguide is used to illuminate the LCOS device. Avengant, in their presentation, also discussed various drive modes of the LEDs to give higher brightness and efficiency with green-only and black-and-white modes. The 13-minute video of Avegant’s presentation is available at the SPIE site (behind SPIE’s paywall). According to Avegant’s presentation, the optics are 15.6mm long by 12.4mm wide, support a 30-degree FOV, with 34 pixels/degree, and 2 lumens of output in full color and up to 6 lumens in limited color outdoor mode. According to the presentation, they expect about 1,500 nits with typical diffractive waveguides in the full-color mode, which would roughly double in the outdoor mode.
The Magic Leap 2 (ML2) takes reducing the optics one step further and puts the illumination LEDs and LCOS on opposite sides of the display’s waveguide (below and described in Magic Leap 2 at SPIE AR/VR/MR 2022). The ML2 claims to have 2,000 nits with a much larger 70-degree FOV.
Transparency (vs. Birdbath) and “Eye Glow”
Transparency
As seen in the pictures above, all the waveguide-based glasses have transparency on the order of 80-90%. This is a far cry from the common birdbath optics, with typically only 25% transparency (see Nreal Teardown: Part 1, Clones and Birdbath Basics). The former Osterhout Design Group (ODG) made birdbath AR Glasses popular first with their R6 and then with the R8 and R9 models (see my 2017 article ODG R-8 and R-9 Optic with OLED Microdisplays) which served as the models for designs such at Nreal and Lenovo’s A3.
OGD Legacy and Progress
Several former ODG designers have ended up at Lenovo, the design firm Pulsar, Digilens, and elsewhere in the AR community. I found pictures of Digilens VP Nima Shams wearing the ODG R9 in 2017 and the Digilens Argo at CES. When I showed the pictures to Nima, he pointed out the progress that had been made. The 2023 Argo is lighter, sticks out less far, has more eye relief,is much more transparent, has a brighter image to the eye, and is much more power efficient. At the same time, it adds features and processing not found on the ODG R8 and R9.
Front Projection (“Eye Glow”)
Another social aspect of AR glasses is Front Projection, known as “Eye Glow.” Most famously, the Hololens 1 and 2 and the Magic Leap 1 and 2 project much of the light forward. The birdbath optics-based glasses also have front projection issues but are often hidden behind additional dark sunglasses.
When looking at the “eye glow” pictures below, I want to caution you that these are random pictures and not controlled tests. The glasses display radically different brightness settings, and the ambient light is very different. Also, front projection is typically highly directional, so the camera angle has a major effect (and there was no attempt to search for the worst-case angle).
In our AWE 2022 Video with SadlyItsBradley, I discussed how several companies, including Dispelix, are working to reduce front projection. Digilens is one of the companies I discussed that has been working to reduce front projection. Lumus’s reflective approach has inherent advantages in terms of front projection. DigiLens Argo (pictures 2 and 3 from the right) have greatly reduced their eye glow. The Vuzix Shield (with the same optics as the Ultralite) has some front projection (and some on my cheek), as seen in the picture below (4th from the left). Oppo appears to have a fairly pronounced front projection, as seen in two short videos (video 1 and video 2)
DigiLens Argo Deeper Look
DigiLens has been primarily a maker of diffractive waveguides, but it has, through the years, made several near-product demonstrations in the past. A few years ago, they when through a major management change (see 2021 article, DigiLens Visit), and with the management came changes in direction.
Argo’s Business Model
I’m always curious when a “component company” develops an end product. I asked DigiLens to help clarify their business approaches and received the following information (with my edits):
Optical Solutions Licensing – where we provide solutions to our license to build their own waveguides using our scalable printing/contactless copy process. Our licensees can design their waveguides, which Digilens’ software tools enable. This business is aimed at higher-volume applications from larger companies, mostly focused on, but not limited to, the consumer side of the head-worn market.
Enterprise/Industrial Products – ARGO is the first product from DigiLens that targets the enterprise and industrial market as a full solution. It will be built to scale and meet its target market’s compliance and reliability needs. It uses DigiLens optical technology in the waveguides and projector and is built by a team with experience shipping thousands of enterprise & Industrial glasses from Daqri, ODG, and RealWear.
Image Quality
As I was familiar with Digilen’s image quality, I didn’t really check it out that much with the ARGO, but rather I was interested in the overall product concept. Over the last several years, I have seen improved image quality, including uniformity and addressing the “eye glow” issue (discussed earlier).
For the type of applications in the “enterprise market” ARGO is trying to serve, absolute image quality may not be nearly as important as other factors. As I have often said, “Hololens 2 proves that image quality for the customers that use it” (see this set of articles discussing the Hololen 2’s poor image quality). For many AR markets, the display information is simple indicators such as arrows, a few numbers, and lines. It terms of color, it may be good enough if only a few key colors are easily distinguishable.
Overall, Digilens has similar issues with color uniformity across the field of view of all other diffractive waveguides I have seen. In the last few years, they have gone from having poor color uniformity to being among the better diffractive waveguides I have seen. I don’t think any diffractive waveguide would be widely considered good enough for movies and good photographs, but they are good enough to show lines, arrows, and text. But let me add a key caveat, what all companies demonstrate are invariably certainly cherry-picked samples.
Field of View (FOV)
While the Argos 30-degree FOV is considered too small for immersive games, for many “enterprise applications,” it should be more than sufficient. I discussed why very large FOVs are often unnecessary in AR in this blog’s 2109 article FOV Obsession. Many have conflated VR emersion with AR applications that need to support key information with high transparency, lightweight, and hands-free. As Professor and decades-long AR advocate Thad Starner pointed out, requiring the eye to move too much causes discomfort. I make this point because a very large FOV comes at the expense of weight, power, and cost.
Key Feature Set
The diagram below is from DigiLen on the ARGO and outlines the key features. I won’t review all the features, but I want to discuss some of their design choices. Also, I can’t comment on the quality of their various features (SLAM, WiFi, GPS, etc.) as A) I haven’t extensively tried them, and B) I don’t have the equipment or expertise. But at least on the surface, in terms of feature set, Argo compares favorably to the Hololens 1 and 2, if having a smaller FOV than the Hololens 2 but with much better image quality.
Most realistic enterprise-use models for AR headsets include significant, if not exclusively, hands-free operation. The basic idea of mounting a display on the user’s head it so they can keep their hands free. You can’t be working with your hands and have a controller in your hand.
While hand tracking cameras remove the need for the physical controller, they do not free up the hands as the hands are busy making gestures rather than performing the task with their hands. In the implementations I have tried thus far, gestures are even worse than physical controllers in terms of distraction, as they force the user to focus on the gestures to make it (barely sometimes) work. One of the most awful experiences I have had in AR was trying to type in a long WiFi password (with it hidden as I typed by asterisk marks) using gestures on a Hololens 1 (my hands hurt just thinking about it – it was a beyond terrible user experience).
Similarly, as I discussed with SadlyItsBradley about Meta’s BCI wristband, using nerve and/or muscle-detecting wristbands still does not free up the hands. The user still has their hands and mental focus slaved to making the wristband work.
Voice control seems to have big advantages for hands-free operation if it can work accurately in a noisy environment. There is a delicate balance between not recognizing words and phrases, false recognition or activation, and becoming too burdensome with the need for verification.
Skull-Gripping “Glasses” vs. Headband or Open Helmet
In what I see as a futile attempt to sort of look like glasses (big ugly ones at that), many companies have resorted to skull-gripping features. Looking at the skull profile (right), there really isn’t much that will stop the forward rotation of front-heavy AR glasses unless they wrap around the lower part of the occipital bone at the back of the head.
Both the ARGO (below left) and Panasonic’s (Shiftall division) VR headsets (right two images below) take the concept of skull-grabbing glasses to almost comic proportions. Panasonic includes a loop for the headband, and some models also include a forehead pad. The Panasonic Shiftall uses pads pressed against the front of the head to support the front, while the ARGO uses an oversized large noise bridge as found on many other AR “glasses.”
ARGO supports a headband option, but they require the ends of the temples with the skull-grabbers temples to be removed and replaced by a headband.
As anyone who knows anything about human factors with glasses knows, the ears and the nose cannot support much weight, and the ears and nose will get sore if much weight is supported for a long time.
Large soft nose pads are not an answer. There is still too much weight on the nose, and the variety of nose shapes makes them not work well for everyone. In the case of the Argo, the large nose pads also interfere with wearing glasses; the nose pads are located almost precisely where the nose pads for glasses would go.
Bussel/Bun on the Back Weight Distribution – Liberating the Design
As was pointed about by Microsoft with their Hololens 2 (HL2), weight distribution is also very important. I don’t know if they were the first with what I call “the bustle on the back” approach, but it was a massive improvement, as I discussed in Hololens 2 First Impressions: Good Ergonomics, But The LBS Resolution Math Fails! Several others have used a similar approach, most notably with the Meta Quest Pro VR (it has very poor passthrough AR, as I discussed in Meta Quest Pro (Part 1) – Unbelievably Bad AR Passthrough). Another feature of the HL2 ergonomics is the forehead pad eliminates weight from the nose and frees up that area in support of ordinary prescription glasses.
The problem with the sort-of-glasses form factor so common in most AR headsets today is that it locks the design into other poor decisions, not the least of which is putting too much weight too far forward. Once it is realized that these are not really glasses, it frees up other design features for improvement. Weight can be taken out of the front and moved to the back for better weight distribution.
ARGO’s Eye-Relief Missed Opportunity for Supporting Normal Glasses
Perhaps the best ergonomic/user feature of the Hololens 1 & 2 over most other AR headsets is that they have enough eye relief (distance from the waveguide to the eye) and space to support most normal eyeglasses. The ARGO’s waveguide and optical design have enough eye relief to support wearing most normal glasses, but still, they require specialized inserts.
You might notice some “eye glow” in the CNET picture (above right). I think this is not from the waveguide itself but is a reflection off of the prescription inserts (likely, they don’t have good anti-reflective coatings).
A big part of the problem with supporting eyeglasses goes back to trying to maintain the fiction of a “glasses form factor.” The nose bridge support will get in the way of the glasses, but the nose bridge support is required to support the headset. Additionally, hardware in the “brow” over the eyes could have been moved elsewhere, which may interfere.
Another technical issue is the location and shape of their optical engine. As discussed earlier, the Digilens engine shape causes issues with jutting into the front of glasses, resulting in a large brow over the eyes. This brow, in turn, may interfere with various eyeglasses.
It looks like Argo started with the premise of looking like glasses putting form ahead of function. As it turns out, they have what for me is an unhappy compromise that neither looks like glasses nor has the Hololens 2 advantage of working with most normal glasses. Starting from the comfort and functionality as primary would have also led to a different form factor for the optical engine.
Conclusions
While MicroLED may hold many long-term advantages, they are not ready to go head-to-head with LCOS engines regarding image quality and color. The LCOS engines are being shown by multiple companies that are more than competitive in size and shape with the small MicroLED engines. The LCOS engines are also supporting much higher resolutions and larger FOVs.
Lumus, with their Z-Lens 2-D reflective waveguides, seems to have a big advantage in image quality and efficiency over the many diffractive waveguides. Allowing the Z-lens to be encased without an air gap adds another significant advantage.
Yet today, most waveguide-based AR glasses use diffractive waveguides. The reasons include there being many sources of diffractive waveguides, and companies can make their own custom designs. In contrast, Lumus controls its reflective waveguide I.P. Additionally, Lumus has only recently developed 2-D reflective waveguides, dramatically reducing the size of the projection engine driving their waveguides. But the biggest reason for using diffraction waveguides is that the cost of Lumus waveguides is thought to be more expensive; Lumus and their new manufacturing partner Schott Glass claimed that they will be able to make waveguides at competitive or better costs.
A combination of cost, color, and image quality will likely limit MicroLEDs for use in ultra-small and light glasses with low amounts of visual content, known as “data snacking.” (think arrows and simple text and not web browsing and movies). This market could be attractive in enterprise applications. I’m doubtful that consumers will be very accepting of monochrome displays. I’m reminded of a quote from an IBM executive in the 1980s when asked whether resolution or color was more important said: “Color is the least necessary and most desired feature in a display.”
Not to pick on Argo, but it demonstrates many of the issues with making a full-featured device in a glasses form factor, as SLAM (with multiple spatially separated cameras), processing, communication, batteries, etc., the overall design strays away from looking like glasses. As I wrote in my 2019 article, Starts with Ray-Ban®, Ends Up Like Hololens.
My coverage of CES and SPIE AR/VR/MR 2023 continues, this time on MicroLEDs. MicroLEDs companies were abundant in the booths, talks, and private conversations at AR/VR/MR 2023.
The list on the right shows some of the MicroLED companies I have looked at in recent years. Marked with a blue asterisk “*” are companies I talked to at AR/VR/MR 2023, with Jade Bird Display (JBD), PlayNitride, Porotech, and MICLEDI having booths in the exhibition. The green bracket on the left indicates companies where I had seen a MicroLED display generating an image (not just one or a few LEDs). Inside the gold rectangle in the list above are MicroLED companies that system companies have bought. MicroLEDs are the display technology where tech giants Meta, Apple, and Google place their bets for the future.
A much more extensive list of companies involved in MicroLED development can be found at microled-info.com, a site dedicated to tracking the MicroLED industry. Microled-info’s parent company, Metalgrass, also organized the MicroLED Association, and I spoke at their Feb. 7th Webinar (but you have to join the association to see it).
The efficiency of getting the Lambertian light that most LEDs emit through a waveguide to the eye is a major issue I have studied for years and will be covered first. Then after covering recent MicroLED prototypes and discussions, I have included an appendix with background information in the subsections “What is a MicroLED company,” “Microdisplay vs. Direct View Pixel Sizes,” and “Multicolor, Full Color, or True Color.”
MicroLEDs and Waveguides; Millions of Nits-In to Thousands of Nits-Out with Waveguides
When first hearing of MicroLEDs outputting millions of nits, you might think it must be overkill to deliver thousands of nits to the eye for outdoor use with a waveguide. But due to pupil expansion and light losses, only a tiny fraction of the light-in makes it to the eye. The figure (right) diagrams the efficiency issues with waveguides using a diffractive waveguide.
Most LEDs output diffuse (roughly) Lambertian light, whereas waveguides require collimated light. Typically, micro-optics such as microlens arrays (MLA) are on top of the MicroLEDs’ semi-collimate the light. These optics increase the nits; typically, the nits quoted for the MicroLED display are after micro-optics. A waveguide’s small entrance area severely limits the light due to a physics property known as “etendue,” causing it to be called “etendue loss.” Then there are the losses due to the pupil expansion/replication structures (diffraction gratings in the case of diffractive waveguides, semi-reflective “facets” in the case of reflective waveguides). Finally, the light-in from the small entrance area ends up spread out over the much larger exit area to support seeing the image over the whole FOV as the eye moves.
Multiple Headsets Using Diffractive Waveguides with JBD MicroLED
I found it an interesting dichotomy that while all the other prototypes I have seen using Jade Bird Display (JBD) MicroLEDs, including Vuzix, Oppo, TCL, Dispelix, and Waveoptics (before being acquired by Snap), JBD themselves showed a prototype 3-chip color cube projector with a Lochn “clone” (with lesser image quality) of a Lumus 2D expanding reflective waveguide in their booth (I was asked not photograph). Then in the Playnitride booth, they featured Lumus reflective waveguides. I should note that while efficiency is a major factor, other design factors, including cost, will drive different decisions.
Reflective (Lumus) Waveguides are More Efficient than Diffractive Waveguides with MicroLEDs
According to Lumus, their 2-D reflective (Lumus) waveguides result in a 3 to 9 times larger entrance area, and their semi-reflective facets lose less light than diffraction gratings.The net result is that reflective waveguides can be 5 to >10 times more optically efficient than diffractive waveguides with the same microLEDs, a major advantage in brightness and power (= less heat and longer battery life). This efficiency advantage appears to have been playing out at AR/VR/MR 2023.
Playnitride prominently showed their MicroLEDs using Lumus 2D and older 1D reflective waveguides in their booth (below left and middle). Their full-color QD-MicroLEDs only output about 150K nits (compared to the millions of others’ single-color native LEDs), so they needed a more efficient waveguide. Playnitride uses Quantum Dot conversion of blue LEDs to give red and green.
Lumus CTO Dr. Yochay Danziger brought a 2D expanding waveguide with input optics that he held up to Porotech’s MicroLEDs. I captured a quick handheld (and thus not very good) shot (with ND filters to reduce the very bright image) of Porotech’s green MicroLED via Lumus’s handheld waveguide (above right).
Lumus was the only company featured in the Schott Glasses booth at AR/VR/MR 2023. The often-asked question about Lumus is whether they can make them in volume production. The Schott Glass representative assured me they could make Lumus’s 2-D waveguides in volume production.
Other Optics (ex., Bird Bath, Freeform, and VR-Pancake) and Micro-OLEDs
I want to note here that while MicroLEDs are hundreds to over a thousand times brighter than Micro-OLEDs, they are likely well more than five years away from having anywhere near the same color control and uniformity. Thus designs that favor image quality over brightness using optical designs that are much more efficient than waveguides, such as Bird Bath, Freeform, and VR-pancake optics, will continue to use Micro-OLEDs or LCDs for the foreseeable future. Micro-OLEDs are expected to continue getting brighter, with some claiming they have roadmaps to get to about 30K nits.
Jade Bird Display (JBD) Based AR Glasses
Jade Bird Display (JBD) is the only company I know to be shipping MicroLEDs in production. All working headsets I have seen use JBD’s 640×480 green (only) MicroLEDs, including ones from Vuzix (Ultralite and Shield), Oppo, and Waveoptics (shown in 2022 before being acquired by Snap). JBD is developing devices supporting higher pixel depth and higher resolution.
The current green MicroLEDs support only 4 bits per pixel or 16 (24) brightness levels and will show contour lines with a smooth shaded area. I hear that JBD’s future designs will support more levels. While I have seen continuous improvement in the pixel-to-pixel brightness differences through the year, and while they are the most uniform MicroLED devices I have seen, there is still visible “grain” in what should be a solid area.
Vuzix
At CES 2023, Vuzix showed off the small size possible with their Utralite™ glasses (left side below) which weigh only 38 grams (not much more than most conventional glass). A tray full of display engines on public display was there to emphasize that they were in production. The comparison of light engines (below left) shows how compact the MicroLED green and color cube projector engines are compared with Vuzix’s older (but true color) DLP design with similar resolution. I discussed Vuzix’s Ultralite and Shield in the CES 2023 video with SadleyItsBradley.
The Vuzix Shield and Ultralite share the same small green MicroLED engine. The combination of the engine and Vuzix waveguide are capable of up to 4,100 nits which is bright enough to enable outdoor use. The power consumption of MicroLEDs is roughly proportional to the average pixel value (APV). Paul Travers, CEO of Vuzix, says that the Ultralites consume very little power and can work for two days in typical use on a charge. Vuzix has also improved their in-house developed waveguides, significantly reducing the forward projection (“eye-glow”).
At AR/VR/MR 2023, Oppo showed me their JBD green MicroLED based glassed with a form factor similar to the Vuzix Ultralite. The overall image quality and resolution seem similar on casual inspection. The Vuzix waveguides diffraction gratings seem less noticeable from the outside, but I have not compared them side by side in the same conditions.
The TCL glasses appear to be using a diffraction grating waveguide that is very different from others I have seen due to the way the exit grating has very big steps in the transmission of light (right). This waveguide differs from the reflective waveguide JBD was showing in their booth or other diffractive waveguides. I have seen diffractive waveguides that were none uniform but never with such large steps in the output gratings. While I didn’t get a chance to see an image through the TCL glasses, the reports I got from others were that the image quality was not very good.
Goertek/Goeroptics Design and Manufacturing JBD Projection Engines
In the CES 2023 TCL video, I discussed some of the issues associated X-Cube color combining and the problems with aligning the three panels. At the AR/VR/MR conference, the Goeroptics division of Goertek showed that they were making both green-only and Color X-Cube designs for JBD’s MicroLEDs (slide from their presentation below). While Goertek may not be a household name, they are a very large optics and not-optics design and OEM for many famous brands, including giants such as Apple, Microsoft, Sony, Samsung, and Lenovo.
Porotech, Ostendo, and Innovation Semiconductor color tunable LEDs
Below is a very short video I captured in the Porotech booth with a macro lens of their DynamicPixelTuning demo. I apologize for the camera changing focus when I switched from still to video mode with the blooming due to the wide range of brightness as the color changes. The demo shows the whole display changing color, as Porotech does not have a backplane that can change colors pixel by pixel.
Porotech showed a combination of motion and color changing with their DynamicPixelTuning
At CES 2023, I was reminded by Ostendo, best known for the color-stacked MicroLEDs technology, that they had developed tunable color LEDs several years ago. Sure enough, six years ago, Ostendo presented the paper III-nitride monolithic LED covering full RGB color gamutin the Journal of the SPIE in February 2016. I have not seen evidence that Ostendo has come close to pursuing it beyond the single LED prototype stage, as Porotech has done with their DynamicPixelTuning.
The recent startup Innovation Semiconductor (below) is developing technology to integrate the control transistor circuitry into the InGaS substrate and avoid the more common hybrid InaS, and CMOS approaches almost all others are using. They are also developing a “V-grove” technology for making color-tunable LEDs. Innovation Semi cites work by the University of California at Stata Barbara (see paper 1 and paper 2 ) plus their own work that suggests that V-groves may be a more efficient way to produce color-tunable LEDs than the approach taken by Porotech and Ostendo.
A major concern I have with Innovation Semi’s approach to integrating the control transistors in GaN is whether they will be able to integrate enough control circuitry without making the devices too expensive and/or making the pixel size bigger.
PlayNitride (Blue with QD Conversion Spatial Color)
PlayNitride demonstrated its full-color MicroLED technology, which uses blue LEDs with Quantum Dot (QD) conversion to produce red and green. At 150K nits, they are extremely bright compared to Micro-OLEDs but are much less bright than native red, green, and blue MicroLEDs from companies including JBD and Porotech.
As discussed earlier, PlayNitride showed their MicroLEDs working with Lumus waveguides. But even though Lumus waveguides are more efficient than diffractive waveguides, 150K nits from the display are not bright enough for practical uses. They are about 1/10th the brightness of the native MicroLEDs of JBD and Porotech, and their pixels are bigger.
PlayNitride was the only company showing fairly high-resolution (1K by 1K and 1080P) full-color single-chip MicroLED microdisplays. Furthermore, these are only prototypes. Still, the green and red were substantially weaker than the blue, as seen in the direct (no waveguide) macro photograph of PlayNitrides MicroLED below. Also, the red was more magenta (mixed red and blue).
Looking at the 2X zoom, one sees the “grain” associated with the pixel-to-pixel brightness differences in all colors common to all MicroLEDs demonstrated to date. Additionally, in the larger reddish wedge pointed at by the red arrow, there are color differences/grain at the pixel level.
Known issue with QD spatial color conversion and microdisplays
While quantum dot (QD) color conversion of blue and UV LEDs has been proposed as a method to make full-color MicroLEDs for many years, there are particular issues with using QD with very small microdisplay pixels. Normally the QD layer required for conversion stays roughly the same thickness as the pixels become smaller, resulting in a very tall stack of QD compared to the pixel size. It then requires some form of microscopic baffling to prevent the light from adjacent LEDs from illuminating the wrong color.
Some have tried using thinner layers of QD and then relied on color filters to “clean up” the colors, but this comes with significant losses in efficiency and issues with heat. There are also issues with how hard the QD material can be driven before it degrades, which will limit brightness. Using spatial color itself has the issue of pixel sizes becoming too big for use in AR.
Many of these issues will be very different for making larger direct-view and VR pixels. The thickness of the QD layers becomes a non-issue as the pixels get bigger and spatial color has long been used with larger pixels. We have already seen where different OLED technologies have been used based on pixel size and application; for example, color-filtered OLEDs won out in large-screen TVs, whereas native color OLED subpixels are used in smartphone phones, smartwatches, and microdisplay OLEDs.
MICLEDI Reconstituted InGaS Wafers
MICLEDI is a spinout of the IMEC research institute in Belgium in 2019 with a booth at AR/VR/MR 2023. They are fabless with a mix of MicroLED technologies they have developed (right). They claim to have single color per die, spatial color (colors side by side), and stacked color technology. They have also developed GaN and Aluminum Gallium Phosphor (AlinGAP) red. After some brief discussions in their booth and going through their handout material, their MicroLEDs seem like a bit of a grab bag of technology for license without a clear direction.
The one technology that seems to set MICLEDI apart is for taking 100, 150mm, or 200mm GaN or AlinGap EPI wafers and making a “reconstituted” wafer with pick and placed known good dies. These reconstituted wafers can be “flip chipped” with today’s 300mm CMOS wafers. Today, almost all LED manufacturing is on much smaller wafers than mainstream production CMOS. For development today, companies are flipping small GaN wafers with spaced-out sets of LED arrays onto a larger CMOS wafer and throwing away most of the CMOS wafer.
Stacked MicroLEDs
While I didn’t see MIT at CES or AR/VR/MR 2023, MIT made news during AR/VR/MR with stacked color MicroLEDs. I don’t know the details, but it sounds similar to what Ostendo discussed, at least as far back as 2016 (see lower left). MICLEDI (above) has also developed a stated color LED technology where the LEDs are side by side.
The obvious advantage of stacked color is that the full color is smaller. But the disadvantage is that the LEDs and other circuitry above block light from lower LEDs. The net result is that stacked LEDs will likely be much brighter than Micro-OLEDs but much less bright than other MicroLED technologies. Also concerning is that while red is the color with the least efficiency today, it seems to end up on the lowest layer.
With their mid-range brightness, stacked MicroLEDs would likely be targeted at non-waveguide optics designs. Ostendo has been developing its optical design, which tiles multiple small MicroLEDs to give a wider FOV.
Conclusions
Many giant and small companies are betting that MicroLEDs will be the future of MicroDisplay technology for VR and AR. At the same time, one should realize that none of the technologies is competitive today regarding image quality with Micro-OLED, LCOS, or DLP. There are many manufacturing and technical hurdles yet to be solved. Each of the methods for producing full-color MicroLEDs has advantages and disadvantages. The race in AR is to support full-color displays and higher resolution at high brightness as, low power, and small size. I can’t see how multiple monochrome displays using X-Cubes, Waveguides, or other methods are long-term AR solutions.
I often warn people that if someone does a demo first, that does not mean they will be in production first. Some technical approaches will yield a hand-crafted one-off demo faster but are not manufacturable. The warning is doubly true when it comes to color MicroLEDs. It is easier to rule out certain approaches than to say which approach or approaches will succeed. For MicroDisplay MicroLEDs used in AR, I think native LEDs will win out over color-converted (ex., QD) blue LEDs. A different MicroLED technology will likely be better for direct-view displays.
It will be interesting to see the market adoptions of the new small form factor but green-only AR glasses. While they meet the form factor requirement of looking like glasses with acceptable weight, they don’t have great vision correction solutions, and being green-only will limit consumer interest.
A continuing issue with be which optics work best with MicroLEDs. Part of this issue will be affected by the degree of collimation of the light from the LEDs. The 2-D reflective waveguides developed by Lumus have a significant efficiency advantage, but still, many more companies are using diffractive waveguides today.
Appendix: MicroLED Background Information
What is a MicroLED Company?
To have a successful MicroLEDs is more than making the LEDs; it is about making a complete display and the ability to control it accurately at an affordable cost.
What constitutes a “MicroLED company” varies widely from a completely fabless design company to one that might design and fab the LEDs, design the (typically) CMOS control backplane, and then do the assembly and electrical connection of the (typically) Indium Gallium Nitride (InGaS) LEDs onto the CMOS backplane. Almost every company has a different “flow” or order in which they assemble/combine various component technologies. For example, shown below is the flow given by JBD, where they appear to be applying the Epi-lay to grow the LEDs on top of the CMOS wafer; other companies would form the LEDs first on the InGaN wafer and then bond the finished transistor arrays onto the finished CMOS control devices.
There is no common approach, and there are as many different methods as there are companies with some flows radically different from JBD’s. Greatly complicating matters is that most InGaN fabrication is done on 150mm to 200mm diameter wafers. In contrast, mainstream CMOS today is made on 300mm wafers which least to a variety of methods to address this issue, some of which are better suited to volume manufacturing than others.
Microdisplay vs. Direct View Pixel Sizes
What companies call MicroLED displays varies from wall-size monitors and TVs that can be more than a meter wide down to microdisplays typically less than 25mm in diagonal. As the table on the right shows, a small pixel on an AR microdisplay is about 300 to 600 times smaller than a direct-view smartphone or smartwatch. Pixel sizes get closer when comparing waveguide-based AR to VR pixels.
VR headsets started with essentially direct-view cell phone-type displays with some cheap optics to enable the human eye to focus but have been driving the pixel size down to improve angular resolution. The latest trend is to use pancake optics which can use even smaller pixels to enable smaller headsets.
There is some “bridging” between AR and VR with display types. For example, large combiner “bug-eye” AR often uses direct-view type displays common in VR. Some pancake optics-based VR displays use the same Micro-OLED displays used with AR birdbath optics.
With the radically different pixel sizes, it should not be surprising that the best technology to support that pixel size could change. Small microdisplays used by waveguide-based AR require microdisplays with semiconductor (usually CMOS) transistors. TVs, smartphones, and smartwatches use various types of thin film transistors.
Particularly regarding supporting color with MicroLEDs, it should be expected that the technologies used for microdisplays could be very different from those used for direct-view type displays. For example, while quantum dots color conversion of blue or UV light might be a good method for supporting larger displays, it does not seem to scale well to the small pixel sizes used in AR.
While not “industry standard definitions,” for the sake of discussion, I want to define three categories of color display:
Multicolor – Provides multiple identifiable colors, including, at a minimum, the primary colors of red, green, and blue. This type of display is useful for providing basic information and color coding it. Photographic images will look cartoonish at best, and there are typically very visible “contour lines” in what should be smoothly shaded surfaces.
Full Color – This case supports a wide range of colors, and smooth surfaces will not have significant contours, but the color control across the display is not good enough for showing pictures of people.
True Color – The display is capable of reasonably accurate color control across the display. Importantly, faces and skin tones, to which human vision is particularly sensitive, look good. It a display is “true color,” it should also be able to control the “white point,” and whites will look white, and grays will be gray. There should be no visible contouring.
The images below are examples of “multicolor,” “full color,” and “true color” images.
JBD “Multicolor” Display
Playnitride “Full Color”
KGOnTech Test Pat. “True Color”
It might seem to some that my definition of “full” versus “true” color is redundant, but I have seen many demonstrations through the years where the display can display color but can’t control it well. In 2012, I wrote Cynics Guide to CES – Glossary of Terms. I called this issue “Pixar-ized” because there were so many demos of cartoon characters showing color saturation but none showing humans, which requires accurate color control.
Pixar-ized – The showing of only cartoons because the device can’t control color well and/or has low resolution. People have very poor absolute color perception but tend to be are very sensitive to skin tones and know what looks right when viewing humans, but the human visual systems is very poor at judging whether the color is right in a cartoon. Additionally it is very hard to tell resolution when viewing a cartoon.
I will add to this category above “artistic” false/shifted color images (see Playnitride’s above). Sometimes this is done because the work to calibrate the prototype has not been completed, even though the display can eventually support full color. Still, it is often done to hide problems.
I should note that what can be acceptable to the eye with a single-color image can look very bad when combined with other colors. What are weak or dead pixels with a monochrome display will turn into colorized or color-shifted pixels that will stick out. Anyone with a single dead color within a pixel on display has seen how the missing color sticks out. The images below are a simplified Photoshop (simulation) of what happens if random noise and dim areas occur in the various colors. The left image shows the effect on the full-color image, and the right image shows the same amount of random noise and dimming (in green) with the monochrome green (note, the image on the right is the grayscale image and then converted to green and not just the green channel from the true color image). In the green-only image, you can see some noise and a slight dimming that might not even be noticeable, whereas, in the color image, it turns into a magenta-colored area.
In that same 2012 article, I wrote about “Stilliphobia,” the fear of showing still images. We are seeing that with displaying content that is very busy and/or with lots of motion to hide dead or weak pixels or random pixel values in the display. When I see a needlessly busy image or lots of motion, I immediately think they are trying to hide problems. Someone with a great-looking display should show pictures of people and smooth images for at least some content.
Most of today’s MicroLED displays are working on getting to multicolor displays and are far from true color. All MicroLED microdisplays I have seen to date have large pixel-to-pixel variations. No amount of calibration or mura correction will be enough to produce a good photographic image if the individual colors can’t be controlled accurately. The good news is that most of today’s AR applications only require a multicolor display.
[March 4th, 2023 Corrections/Updates – poLight informed me of some corrections, better figures, and new information that I have added to the section on poLight. Cambridge Mechatronics informed me about their voltage and current requirements for pixel-shifting (aka wobulation).]
Introduction
For this next entry in my series on companies I met with at CES or Photonics West’s (PW) AR/VR/MR show in 2023, I will be covering two different approaches to what I call “optics micromovement.” Cambridge Mechatronics (CML) uses Shape Memory Alloys (SMA) wires to move optics and devices (including haptics). poLight uses piezoelectric actuators to bend thin glass over their flexible optical polymer. I met with both companies at CES 2023, and they both provided me with some of their presentation material for use in this article.
After discussing the technologies from CML and poLight, it will be got into some of the new uses within AR and VR.
Beyond Camera Focusing and Optical Image Stabilization Uses of Optics Micromovement in AR and VR
Both poLight and CML have cell phone customers using their technology for camera auto-focus and optical image stabilization (OIS). This type of technology will also be used in the various cameras found on AR and VR headsets. poLight’s TLens is known to be used in the Magic Leap 2 reported by Yole Development and Sharp’s CES 2023 VR prototype (reported by SadlItsBradley).
While the potential use of their technology in AR and VR camera optics is obvious, both companies are looking at other ways their technologies could support Augmented and Virtual Reality.
Cambridge Mechatronics (CML) – How it works
Cambridge Mechatronics is an engineering firm that makes custom designs for miniature machines using shaped memory alloy (SMA). Their business is in engineering the machines for their customers. These machines can move optics or objects. The SMA wires contract when heated due to electricity moving through them (below left) and then act on spring structures to cause movement as the wires contract or relax. Using multiple wires in various structures can cause more complex movement. Another characteristic of the SMA wire is that as it heats and contracts, it makes the wire thicker and shorter, causing the resistance to be reduced. CML uses the change in resistance as feedback for closed-loop control (below right).
Show (below right) is a 4-wire actuator that can move horizontally, vertically, or rotate (arrows pointing at the relaxed wires). The SMA wires enable a very thin structure. Below is a still from a CML video showing this type of actuator’s motion.
Below is an 8-wire (2 crossed wires on four sides) mechanism for moving a lens in X, Y, and Z and Pitch and Yaw to control focusing and optical image stabilization (OIS). Below are five still frames from a CML video on how the 8-wire mechanism works.
CML is developing some new SMA technology called “Zero Hold Power.” With this technology, they only need to apply power when moving optics. They suggest this technology would be useful in AR headsets to adjust for temperature variations in optics and support vergence accommodation conflict.
CML’s SMA wire method makes miniature motors and machines that may or may not include optics. With various configurations of wires, springs, levers, ratcheting mechanisms, etc., all kinds of different motions are possible. The issue becomes the mass of the “payload” and how fast the SMA wires can respond.
CML expects that when continuously pixel shifting, they will use take than 3.2V at ~20mA.
poLight – How It Works
poLight’s TLens uses piezoelectric actuators to bend a thin glass membrane over poLight’s special optical clear, index-matched polymer (see below). This bending process changes the lens’s focal point, similar to how the human eye works. The TLens can also be combined with other optics (below right) to support OIS and autofocus.
The GIF animation (right) show how the piezo actuators can bend the top glass membrane to change the lens in the center for autofocus, tilt the lens to shift the image for OIS, and both perform autofocus and OIS.
poLight also proposes supporting “supra” resolution (pixel shifting) for MicroLEDs by tilting flat glass with poLight’s polymer using piezo actuators to shift pixels optically.
One concern is that poLight’s actuators require up to 50 Volts. Generating higher voltages typically comes with some power loss and more components. [Corrected – March 3, 2023]poLight’s companion driver ASIC (PD50) has built-in EMI reduction that minimizes external components (it only requires ext. capacitive load) and power/current consumption is kept very low (TLens® being an optical device, consumes virtually no power, majority of <6mW total power is consumed by our driver ASIC – see table below).
poLight says that the TLens is about 94% transparent. The front aperture diameter of the TLens, while large enough for small sensor (like a smartphone) cameras, seems small at just over 2mm. The tunable wedge concept could have a much wider aperture as it does not need to form a lens. While the poLight method may result in a more compact design, the range of optics would seem to be limited in both the size of the aperture and how much the optics change.
Uses for Optics Micromovement in AR and VR beyond cameras
Going beyond the established camera uses, including autofocus and OIS, outlined below are some of the uses for these devices in AR and VR:
Variable focus, including addressing vergence accommodation conflict (VAC)
Super-resolution – shifting the display device or the optic to improve the effective resolution
Aiming and moving cameras:
When doing VR with camera-passthrough, there are human factor advantages to having the cameras positioned and aimed the same as the person’s eyes.
For SLAM and tracking cameras, more area could be covered with higher precision if the cameras rotate.
Shifting several LEDs to the same location to average their brightness and correct for any dead or weak pixels should greatly improve yields.
Shifting spatial color subpixels (red, green, and blue) to the same location for a full-color pixel. This would be a way to reduce the effective size of a pixel and “cheat” the etendue issue caused by a larger spatial color pixel.
Improve resolution as the MicroLED emission area is typically much smaller than the pitch between pixels. There might be no overlap when switching and thus give the full resolution advantage. This technique could provide even fewer pixels with fewer connections, but there will be a tradeoff in maximum brightness that can be achieved.
Conclusions
It seems clear that future AR and VR systems will require changing optics at a minimum for autofocusing. There is also the obvious need to support focus-changing optics for VAC. Moving/changing optics will find many other uses in future AR and VR systems.
Between poLight and Cambridge Mechatronic (CML), it seems clear that CML’s technology is much more adaptable to a wider range and types of motion. For example, CML could handle the bigger lenses required for VAC in VR. poLight appears to have an advantage in size for small cameras.
This next entry in my series on companies I met with at CES or Photonics West’s (PW) AR/VR/MR show in 2023 will discuss a company working on a headset for a specific application, namely firefighting and related first responders. In discussing Longan Vision, I will mention ThermalGlass (by 360world using Vuzix Blaze optics), Campfire 3D, iGlass, and Mira, which have some similar design features. In addition to some issues common with all AR devices, Longan Vision has unique issues related to firefighting and other first responder applications.
This was my first meeting with Longan Vision, and it was not for very long. I want to be clear that I have no experience working with firefighters or their needs and opinions on AR equipment. In this short article, I want to point out how they tried to address the user’s needs in an AR headset.
Longan Vision
Below is a picture of Longan Vision’s booth, my notations, and some inset pictures from Longan’s website.
Hands-free operation is a big point and central to the use case for many AR designs. Longan uses AR to enhance vision by letting firefighters see through the smoke and darkness and providing additional life-saving information such as temperature and direction.
The AR optics are one of the simplest and least expensive possible; they use dual merged large curved free-space combiners, often called “bug-eye” combiners based on their appearance. They use a single cell phone-size display device to generate the image (some bug-eyes use two smaller displays). The combiner has a partial mirror coating to reflect the display’s image to the eye. The curvature of the semi-reflective combiner magnifies and moves the focus of the display, while light from the real world will be dimmed by roughly the amount of the display’s light reflected.
The combiner is inexpensive to produce with reasonably good image quality. This means it can also be replaced inexpensively if it becomes damaged.
It gives very large eye relief, so there are no issues with wearing glasses. Thus it can be worn interchangeably by almost everyone (one size fits all).
It is optically efficient compared to Birdbath, Waveguides, and most other AR optics.
While large, the combiner can be made out of very rugged plastics and is not likely to break and will not shatter. It can even serve as eye and face protection.
Where the eyes will verge is molded into the optics and will differ from person to person based on their IPD.
As the name “bug-eye” suggests, they are big and unattractive.
Because the combiner magnifies a very large (by near-eye standards) display with very large pixels, the angular resolution (pixels per degree) is very low, while the FOV is large.
Because the combiner is “off-axis” relative to the display, the magnification and focus are variable. This effect can be reduced but not eliminated by making the combiner aspherical. Birdbath optics (described here and shown above-right) have a beamsplitter, which greatly reduces efficiency but makes optics “on-axis” to eliminate these issues.
Brightness is limited by the display’s brightness multiplied by the fraction of light reflected by the combiner. Typically, flat panels will have between 500 and 1,000 nits. That fraction typically ranges between 50% and 20% depending on the tradeoff of display efficiency versus transparency of the real world. These factors and others typically limit their use of indoor applications.
Longan also had some unique requirements incorporated into their design:
The combiner had to be made out of high-temperature plastics
They had to use high-temperature batteries, which added some weight and bulk. Due to their flammability, they could not use the common, more energy-dense lithium batteries.
The combiner supports flipping up to get out of the user’s vision. This is a feature supported by some other bug-eye designs.
The combiner also acts as an eye and partial face shield. Their website demonstration video shows firefighters having an additional flip-up outer protective shield. It is not clear if these will interfere with each other when flipping up and down.
The combiner must accommodate the firefighting breathing apparatus.
An IR camera feeds the display to see what would otherwise be invisible.
Companies with related technologies
I want to mention a few companies that have related technologies.
At CES 2023, I met with ThermalGlass (by 360world), which combined infrared heat images with Vuzix blade technology to produce thermal vision AR glasses. I discussed ThermalGlass in my CES recap with SadlyItsBradley.
Mira has often been discussed on this blog as an example of a low-cost AR headset. Mira’s simple technology is most famously used in Universal Studios Japan, and Hollywood Mario Kart rides. Mira’s website shows a more industrially oriented product with a hard hat and an open frame/band version. Both, like Longan, support a flip-up combiner. The open headband version does not appear to have enough support, with just a headband and forehead pad. Usually, an over-the-head band is also desirable for comfort and a secure fit with this type of support.
The images below show some pictures I took at AWE 2018 of the iView prototype with a large off-axis combiner with a front view (upper left), a view directly of the displays (lower left), and a view through the combiner without any digital correction (below right). The football field in the picture below right illustrates how the image is distorted and how the focus varies from the top to the bottom of the display (the camera was focused at about the middle of the image). Typically the distortion can be corrected in software with some loss in resolution due to the resampling. The focusing issue, however, cannot be corrected digitally and relies on the eye to adjust focus depending on where the eye is centered.
Conclusions
Longan has thought through many features from the firefighter’s user perspective. In terms of optics, it is not the highest-tech solution, but it may not need to be for the intended application. The alternative approach might be to use a waveguide much closer to the eye but with enough eye relief to support glasses. But then the waveguide would have to be extremely ruggedized with its own set of issues in a firefighter’s extreme environment.
Unlike many AR headsets that have me scratching my head. With Longan Vision, I can see the type of customer that might want this product.
Bradley Lynch of the SadleyItsBradley YouTube channel hosted my presentation about CES 2023. The video was recorded about a week after CES, but it took a few weeks to edit and upload everything. There are over 2 hours of Brad and me talking about things we saw at CES 2023.
Brad was doing his usual YouTube content: fully editing the video, improving the visual content, and breaking the video down into small chunks. But it took Brad weeks to get 3 “sub videos” (part 1, part 2, and part 3) posted while continuing to release his own content. Realizing that it would take a very long time at this rate, Brad released part 4 with the rest of the recording session with only light editing as a single 1-hour and 44-minute video with chapters.
For those that follow news about AR and VR, Brad got involved in a controversy with his leaks of information about the Meta Quest Pro and Meta Quest 3. The controversy occurred between the recording and the release of the videos, so I felt I should comment on the issue.
Videos let me cover many more companies
This blog has become highly recognized in the AR/MR community, and I have many more companies wanting me to write about their technology than I have the time. I also want to do in-depth articles, including major through-the-optics studies on “interesting” AR/MR devices.
I have been experimenting with ways to get more content out quicker. I can spend from 3 days to up to 2 months (such as the rest of the Meta Quest Pro series yet to be published) working on a single article about a technology or product. With CES and the AR/VR/MR conference only 3 weeks apart and meeting with about 20 companies at each conference.
In the past, I only had time to write about a few companies that I thought had the most interesting technology. For the CES 2023 video, It took about 3 days to organize the photos and then about 2.5 hours to discuss about 20 companies and their products, or about 5 to 7 minutes per topic (not including all the time spent by Brad doing the video editing).
I liked working with Brad; we hope to do videos together in the future; he is fun to talk to and adds a different perspective with his deep background in VR. But in retrospect, less than half of what we discussed fits with his primary VR audience.
Working on summary articles for CES and the SPIE AR/VR/MR conference
Over 2 hours of Brad and I discussing over 20 companies and various other subjects and opinions about the AR, VR, and MR technology and industry is a lot for people to go through. Additionally, the CES video was shot in one sitting non-stop. Unfortunately, my dog friends decided they wanted to see me in my closed office door closed and barked much more than I realized as I was focused on the presentation (I should have stopped the recording and quieted them down).
I’m working on a “quick take” summary guide with pictures from the video and some comments and corrections/updates. I expect to break the guide into several parts based on broad topics. It might take a few days before this guide gets published as there is so much material.
Assuming the CES quick take guide goes well, I plan to follow up with my quick takes on the AR/VR/MR conference. I’m also looking at recording a discussion at the AR/VR/MR conference that will likely be published on the KGOnTech YouTube channel.
Links to the Various Sections of the Video
Below is a list of topics with links for the four videos.
Between the time of recording the CES 2023 video with Brad and the videos being released, there was some controversy involving Brad and Meta that I felt should be addressed because of my work with Brad.
Brad Lynch made national news when the Verge reported that Meta had caught Brad’s source for the Meta Quest Pro and Meta Quest 3 information and diagrams. Perhaps ironically, the source for the Verge article was a leaked memo by Meta’s CTO, Andrew Bosworth (who goes by Boz). According to The Verge, “In his post to Meta employees, Bosworth confirmed that the unnamed leaker was paid a small sum for sharing the materials with Lynch.“
From what was written in The Verge article and Brad’s subsequent Twitter statement, it seems clear that Brad didn’t know that in journalism is considered unethical “checkbook journalism” to pay a source. It is one of those gray areas where, as I understand it (and not legal advice), it is not illegal unless the reporter is soliciting the leak. At the same time, if I knew Brad was going to pay a source, I would have advised him not to do it.
It is nice to know that news media that will out and out lie, distort, hide key information, and report as true information from highly biased named and unnamed sources still has one slim ethical pillar: leaks are our life’s blood but don’t get caught paying for one. It is no wonder public trust in the news media is so low.
The above said, and to be clear, I never have and would never pay a source or encourage anyone to leak confidential content. I also don’t think it was fair or right for a person under NDA to leak sensitive information except in cases of illegal or dangerous activity by the company.
KGOnTech (My) Stance on Confidentiality
Unless under contract with a significant sum of money, I won’t sign an NDA, as it means taking on a legal and, thus, financial risk. At the same time, when I meet privately with companies, I treat information and material as confidential, even if it is not marked as such, unless they want me to release it. I’m constantly asking companies, “what of this can I write about.”
My principle is that I never want to be responsible for hurting someone that shared information with me. And as stated above, I would never encourage, no less pay someone to break a confidence. If someone shares information with me to publish, I always try to know if they want their name to be public as I don’t want to either get them in trouble or take credit for their effort.
Closing
That’s it for this article. I’ve got to finish my quick take summaries on CES and the AR/VR/MR conference.
Brad Lynch of the SadlyItsBradley YouTube Channel and I sat down for over 2 hours a week after CES and recorded our discussion of more than 20 companies one or both of us met with at CES 2023. Today, Jan. 26, 2023, Brad released a 23-minute part 1 of the series. Brad is doing all the editing while I did much of the talking.
Brad primarily covers VR, while this blog mostly covers optical AR/MR. Our two subjects meet when we discuss “Mixed Reality,” where the virtual and the real world merge.
Brad’s title for part 1 is “XR at CES: Deep Dives #1 (Magic Leap 2, OpenBCI, Meta Materials).” While Brad describes the series as a “Deep Dive,” but I, as an engineer, consider it to be more of an “overview.” It will take many more days to complete my blog series on CES 2023. This video series will briefly discuss many of the same companies I plan to write about in more detail on this blog, so consider it a look ahead at some future articles.
Brad’s description of Part 1 of the series:
There have been many AR/VR CES videos from my channel and others, and while they gave a good overview of the things that could be seen on the show floor and in private demoes, many don’t have a technical background to go into how each thing may work or not work
Therefore I decided to team up with retired Electrical Engineer and AR skeptic, Karl Guttag, to go over all things XR at CES. This first part will talk about things such as the Magic Leap 2, Open BCI’s Project Galea, Meta Materials and a few bits more!
Brad also has broken the video into chapters by subject:
Login
