Introduction: Challenge from Bosworth Accepted

Several people pointed me to an interesting Instagram video AMA (ask me anything) by Meta CTO Andrew Bosworth on October 21, 2024, that appeared to challenge my October 6th article, Meta Orion AR Glasses (Pt. 1 Waveguide), which discussed both transparency and “Eye Glow” (what Bosworth Referred to as “Blue Glow”) — Challenge Accepted.

On the right is a Google Search for “Meta” [and] “Orion” [and] “Eye Glow” OR “Blue Glow” from Sept 7th (Orion announced) through Oct 28, 2024. Everything pertinent to the issue was from this blog or was citing this blog. A Google Search for “Meta Orion” and “blue glow” returns nothing. Shown on the right is a Google search.

As far as I can find, this blog (and a few other sites citing this blog) has been the only one reporting on Meta Orion’s transparency or Eye Glow. So when Bosworth said, “Another thing that was kind of funny about the reviews is people were like, oh, you know you can see the blue glow well,” who else could he be referring to?

Housekeeping – Parts 2 and 3 of the Snap and Orion Roundtable are to be released soon.

The rest of the two-hour roundtable discussion about Snap Spectacles and Meta Orion should be released soon. Part 2 will focus on Meta Orion. Part 3 will discuss more applications and market issues, along with some scuttlebutt about Meta’s EMG wristband controller.

Bosworth’s Statement on Transparency and Eye Glow in Instagram AMA Video – Indirect Shout Out to this Blog

Below is computer transcription with minor light to clean up the speech-to-text and add punctuation and capitalization) of Bosworth’s October 21, 2024, AMA on Instagram, starting at about 14:22 into the video.

14:22 Question: What % of light does Orion block from your view of the world, how much is it darkened?

I don’t know exactly. So, all glass limits transmission to some degree. So, even if you have completely clear glasses, you know, maybe they take you from 100% transmission up your eyes like 97% um, and normal sunglasses that you have are much darker than you think they’re like 17% transmissive is like a standard for sunglasses.  Orion is clear. It’s closer [to clear], I don’t know what the exact number is, but it’s closer to regular prescription glasses than any kind of glasses [in context, he sounds like he is referring to other AR glasses]. There’s no tint on it [Orion]. We did put tint on a couple of demo units so we could see what that looked like, but that’s not how they [Orion] work.

I won’t get into the electrochromic and that kind of stuff.  Some people were theorizing that they were tinted to increase contrast. This is not uncommon [for AR] glasses. We’re actually quite proud that these were not. If I was wearing them, and you’re looking at my eyes, you would just see my eyes.

Note that Bosworth mentioned electrochromic [dimming] but “won’t get into it.” As I stated in Orion Part 1, I believe Orion has electrochromic (electrically controlled) dimming. While not asked, Bosworth gratuitously discusses “Blue Glow,” which in context can only mean “Eye Glow.”

Another thing that was kind of funny about the reviews is people were like, oh, you know you can see the blue glow well. What we noticed was so funny was the photographers from the press who were taking pictures of the glasses would work hard to get this one angle, which is like 15 degrees down and to the side where you do see the blue glow.  That’s what we’re actually shunting the light to. If you’re standing in front of me looking at my eyes, you don’t see the glow, you just see my eyes. We really worked hard on that we’re very proud of it.

But of course, if you’re the person who’s assigned by some journalist outfit to take pictures of these new AR glasses, you want to have pictures that look like you can see something special or different about them. It was so funny as every Outlet included that one angle. And if you look at them all now, you’ll see that they’re all taken from this one specific down and to the side angle.

As far as I can find (it’s difficult to search), this blog is the only place that has discussed the transparency percentage of Orion’s glasses (see: Light Transmission (Dimming?)). Also, as discussed in the introduction, this blog is the only one discussing eye glow (see Eye Glow) in the same article. Then, consider how asking about the percentage of light blockage caused Bosworth to discuss blue [eye] glow — a big coincidence?

But what caused me to write this article is the factually incorrect statement that the only place [the glow] is visible is from “15 degrees down and to the side.“ He doth protest too much, methinks. Most graciously

Orion’s Glow is in pictures taken from more than “taken from this one specific down and to the side angle”

To begin with, the image I show in Meta Orion AR Glasses (Pt. 1 Waveguide), shows a more or less straight-on shot from a video by The Verge (right). It is definitely not shot from a “down and to the side angle.”

In fact, I was able to find images with Bosworth in which the camera was roughly straight on, from down and to the side, and even looking down on the Orion glasses Bosworth’s Sept. 25, 2024, Instagram video and in Adam Savage’s Tested video (far right below).

In the same The Verge Video, there is eye-glow with Mark Zuckerburg looking almost straight on into the camera and from about eye level to the side.

The eye glow was even captured by the person wearing another Orion headset when playing a pong-like game. The images below are composites of the Orion camera and what was shown in the glasses; thus, they are simulated views (and NOT through the Orion’s waveguide). The stills are from The Verge (left) and CNBC (right).

Below are more views of the eye-glow (mostly blue in this case) from the same The Verge video.

The eye glow stills frames below were captured from a CNBC video.

Here are a few more examples of eye glow that were taken while playing the pong-like game from roughly the same location as the CNBC frames above right. They were taken from about even with the glasses but off to the side.

In summary, there is plenty of evidence that the eye glow from Meta’s Orion can be seen from many different angles, not just from below but also from the side, as Bosworth states.

Meta Orion’s Transparency and Electrochromic Dimming

Bosworth’s deflection on the question of Orion’s light transmission

Bosworth started by correctly saying that nothing manmade is completely transparent. A typical (uncoated) glass reflects about 8% of the light. Eyeglasses with good antireflective coatings reflect about 0.5%. The ANSI/ISEA Z87.1, safety glasses standard, specifies “clear” as >85% transmission. Bosworth appears to catch himself knowing that there is a definition for clear and says that Orion is “closer to clear” than sunglasses at about 17%.

Bosworth then says there is “no tint” in Orion, but respectfully, that was NOT the question. He then says, “I won’t get into the electrochromic and that kind of stuff,” which is likely a major contributor to the light transmission. Any dimming technology I know of is going to block much more light than a typical waveguide. The transparency of Orion is a function of the waveguide, dimming layer, other optics layers, and inner and outer protection covers.

Since Bosworth evaded answering the question, I will work through it and try to get an answer. The process will include trying to figure out what kind of dimming I think Orion uses.

What type of electrochromic dimming is Orion Using?

First, I want to put in context what my first article was discussing regarding Orion’s Light Transmission (Dimming?). I was well aware that diffractive waveguides, even glass ones, alone are typically about 85-90% transmissive. From various photographs, I’m pretty sure Orion has some form of electrochromic dimming, as I stated in the first article. I could see the dimming change in one video, and in view of the exploded parts, there appeared to be a dimming device. In looking at this figure, the dimming device seems fairly transparent and on the order of the waveguides and other flat optics. What I was trying to figure out was whether they were using more common polarization-based dimming or a non-polarization-based technology. This picture is inconclusive as to the type of dimming that is used, as the dimmer identified (by me) might be only the liquid crystal part of the shutter with the polarizers, if there are any, in the cover glass or not shown.

The Magic Leap 2 (see: Magic Leap 2 (Pt. 3): Soft Edge Occlusion, a Solution for Investors and Not Users). Polarization-based dimming is fast and gives a very wide range of dimming (from 10:1 to >100:1), but it requires the real-world light first to be polarized, and when everything is considered, it blocks more than 70% of the light. It’s also possible to get somewhat better transmission by using semi-polarizing polarizers, but it gives up a lot of dimming range to gain some transmission. Polarization also causes issues when looking at LCDs, such as computer monitors and some cell phones.

Non-polarization dimming (see, for example, CES & AR/VR/MR Pt. 4 – FlexEnable’s Dimming, Electronic Lenses, & Curved LCDs) blocks less light in its most transmissive state but has less of a dimming range. For example, FlexEnable has a dimming cell that ranges between ~87% transmissive to 35% or less than a 3:1 dimming range. Snap Spectacles 5 uses (based on a LinkedIn post that has since been removed) a non-polarization-based electrochromic dimming by Alphamicron, what they call e-Tint. Both AlphaMicron’s e-Tint and FlexEnable’s dimming use what is known as  Guest-Host LC, which absorbs light rather than changing polarization.

Assuming Orion uses non-polarization dimming, I would assume that the waveguide and related optical surfaces have about 85-90% transmissivity and about 70% to 80% for non-polarization dimming. Since the two effects are multiplicative, that would put Orion in the 90%x80% = 72% to 85 x70% = 60% range.

Orion’s Dimming

Below are a series of images from videos by CNET, The Verge, and Bloomberg. Notice that CNET’s image appears to be much more transmissive. On both CNET and The Verge, I included eye glow pictures from a few frames in the video later to prove both glasses were turned on. CNET’s Orion glasses are significantly more transparent than any other Orion video I have seen (from over 10 I have looked at to date), even when looking at the same demos as in the videos. I missed this big difference when preparing my original article and only discovered it when preparing this article.

Below are some more frame captures on the top row. On the bottom row, there are pictures of the Lumus Maximus (the most transparent waveguide I have seen), WaveOptic Titan, The Magic Leap One (with no tint), and circular polarizing glasses for comparison. The circular polarizing glasses are approximately what I would expect if the Orion glasses were using polarizing dimming.

Snap Spectacles 5, which uses non-polarization dimming, is shown on the left. It compares reasonably well to the CNET mage. Based on the available evidence, it appears that Orion must also be using an electrochromic dimming technology. Per my prior estimate, this would put Orion’s best-case (CNET) transparency in the range of 60-70%

What I don’t know is why CNET was so much more transparent than the others, even when they appear to be in similar lighting. My best guess is that the dimming feature was adjusted differently or disabled for the CNET video.

Why is Orion Using Electronic Dimming Indoors?

All the Orion videos I have seen indicate that Orion is adding electrochromic dimming when indoors. Even bright indoor lighting is much less bright than sunlight. Unlike Snap Spectacles 5 (with electronic dimming) demos, Meta didn’t demo the unit outdoors. There can be several reasons, including:

• The most obvious reason is the lack of display brightness.
• For colors to “pop,” they need to be at least 8x brighter than the surroundings. Bright white objects in a well-lit room could be more than 50 nits. Maybe they couldn’t or didn’t want to go that bright for power/heat reasons.
• Reduced heat of the MicroLEDs
• Saves on battery life

Thinking about this issue made me notice that the walls in the demo room are painted a fairly dark color. Maybe it was a designer’s decision, but it also goes to my saying, “Demos is a Magic Show,” and that darker walls would make the AR display look better.

When this is added up, it suggests that the displays in the demos were likely outputting about 200 nits (just an educated guess). While ~200 nits would be a bright computer monitor, colors would be washed out in a well-lit room when viewed against a non-black background (monitors “bring their own black background”). Simply based on how they demoed it, I suspect that Snap Spectacles 5 is four to eight times brighter than Orion with the dimming used to work outdoors (rather than indoors).

Conclusion and Comments

When I first watched Bosworth’s video, his argument that the eye glow could only be seen from one angle seemed persuasive. But then I went back to check and could easily see that what he stated was provably false. I’m left to speculate as to why he brought up the eye glow issue (as it was not the original question) and proceeded to give erroneous information. It did motivate me to understand Orion better😁.

Based on what I saw in the CNET picture and what is a reasonable assumption for the waveguide, non-polarizing dimmer, and other optics (with transparency being multiplicative and not additive), it pegs Orion in the 60% transparency range plus or minus about 5%.

Bosworth’s answer on transparency was evasive, saying there was no “tint,” which was a non-answer. He mentioned electrochromic dimming but didn’t say for sure that Orion was using it. In the end, he said Orion was closer to prescription glasses (which are about 90% uncoated, 99.5% with anti-reflective coatings) than sunglasses at 17%. If we take uncoated glasses at 90% and sunglasses at 17%, then the midpoint between them would be 53% so that Orion may be, at best, slightly closer to uncoated eyeglasses than sunglasses. There are waveguide-based AR glasses that are more transparent (but without dimming) than Orion.

Bosworth gave more of an off-the-cuff AMA and not a formal presentation for a broad audience, and some level of generalization and goofs are to be expected. While he danced around the transparency issue a bit, it was the “glow” statement and its specificity that I have more of an issue with.

Even though Bosworth is the CTO and head of Meta’s Reality Labs, his background is in software, not optics so that he may have been ill-informed rather than deliberately misleading. I generally find him likable in the videos, and he shares a lot of information (while I have met many people from Meta’s Reality Labs, I have not met Bosworth). At the same time, it sounds to my ear that when he discusses optics, he is parroting things things he has been told, sometimes without fully understanding what he is saying. This is in sharp contrast to, say, Hololen’s former leader, Alex Kipman, who I believe out and out lied repeatedly.

Working on this article caused me to reexamine what Snap Spectacles was using for dimming. In my earlier look at AlphaMicron, I missed that AlphaMicron’s “e-Tint®” was a Guest Host dimming technology rather than a polarization-based one.

From the start, I was pretty sure Orion was using electrochromic dimming, but I was not sure whether it was polarization or non-polarization-based. In working through this article, I’m now reasonably certain it is a non-polarization-based dimming technology.

Working through this article, I realized that the available evidence also suggests that Orion’s display is not very bright. I would guess less than 200 nits, or at least they didn’t want to drive it brighter than that for very long.

Appendix: Determining the light blocking from videos is tricky

Human vision has a huge dynamic range and automatically adjusts as light varies. As Bosworth stated, typical sunglasses are less than 75% transmissive. Human perception of brightness is somewhat binary logarithmic. If there is plenty of available light, most people will barely notice a 50% dimming.

When wearing AR glasses, a large percentage (for some AR headsets, nearly all) of the light needed to view the eye will pass through the AR lens optics twice (in and back out). Because light blocking in series is multiplicative, this can cause the eyes to look much darker than what the person perceives when looking through them.

I set up a simple test using Wave Optic’s waveguide, which is ~85% transmissive, circular polarizing glasses (for 3-D movies) that was 33% tranmissive, and a Magic Leap One waveguide (out of the frame) that was 70% transmissive. In the upper right, I have shown a few examples of where I had a piece of white paper far enough away from the lens that the lens did not affect the illumination of the paper. On the lower right, I moved the paper up against the lens so the paper was primarily illuminated via the lens to demonstrate the light-blocking squared effect.

Orion’s Silicon Carbide (SiC) is not significantly more transparent than glass. Most of the light blocking in a diffraction waveguide comes from the diffraction grating, optical coatings, and number of layers. Considering that Orion’s “hero prototype” with $5B in R&D expenses for only 1,000 units, it is probably more transparent by about 5%.

When looking at open glasses like Orion (unlike, say, Magic Leap or Hololens), the lenses block only part of the eye’s illumination, so you get something less than the square law effect. So, in judging the amount of light blocking, you also have to estimate how much light is getting around the lenses and frames.

Introduction

On October 17th, 2024, Jason McDowell (The AR Show), Jeri Ellsworth (Tilt Five), David Bonelli (Pulsar), Bradley Lynch (SadlyItsBradley), and I recorded a 2-hour roundtable discussion about the recent announcements of the Snap Spectacles 5 and Meta Orion optical AR/MR glasses. Along the way, we discussed various related subjects, including some about the Apple Vision Pro.

I’m breaking the video into several parts to keep some discussions from being buried in a single long video. In this first part, we primarily discuss the Snap Spectacles 5 (SS5). The SS5 will be discussed some more in the other parts, which will be released later. We also made some comments on the Apple Vision Pro, which Bradley Lynch and I own.

The 2-hour roundtable is being released in several parts, with AR Roundtable Part 1 Snap Spectacles 5 and some Apple Vision Pro being the first to be released.

0:00 Introduction of the panelist

Jason McDowall, as moderator, gets things going by having each panelist introduce themself.

2:11 See-Through versus Passthrough Mixed Reality

I gave a very brief explanation of the difference between see-through/optical AR/MR and passthrough MR. The big point is that with See-through/Optical AR/MR, the real world’s view is most important, and with passthrough MR, the virtual world is more important. With passthrough MR, the virtual world is most important with the camera’s view augmenting the virtual world.

5:51 Snap Spectacles 5 (SS5) experience and discussion

Jason McDowell had the opportunity to get a demo of the Snap Spectacles 5 and followed by a discussion by the panelist. Jason has a more detailed explanation of his experience and an interview with Sophia Dominguez, the Director of AR Platform Partnerships and Ecosystem at Snap, on his podcast.

11:59 Dimming (light blocking) with optical AR glasses

Jason noted the dimming feature of the SS5, and this led to the discussion of the need for light blocking with see-through AR.

19:15 See-though AR is not well suited for watching movies and TV

I make the point that see-through AR is not going to be a good device for watching movies and TV.

19:54 What is the application?

We get into a discussion of the applications for see-through AR

20:23 Snap’s motivation? And more on applications

There is some discussion about what is driving Snap to make Spectacles, followed by more discussion of applications.

22:35 What are Snap’s and Meta’s motivations?

The panelist gives their opinions on what is motivating Snap and Meta to enter into the see-through AR space.

23:31 What makes something “portable?”

David makes the point that if AR glasses are not all-day wearable, then they are not very portable. When you take them off, you have fragile things to protect in a case that is a lot bigger and bulkier than a smartphone you can shove in your pocket.

24:13 Wearable AI (Humane AT and Rabbit)

Many companies are working on “AI wearable” devices. We know many companies are looking to combine a small FOV display (typically 25-35 degrees) with audio “AI” glasses.

24:40 Reviewers/Media Chasing the Shiney Object (Apple Vision Pro and Meta Orion)

25:45 Need for a “$99 Google Glass”

Jeri liked Google Glass and thinks there is a place for a “$99 Google Glass”-like product in the market. David adds some information about the economics of ramping up production of the semi-custom display that Google Glass uses. I (Karl) then discuss some of the ecosystem issues of making a volume product.

27:28 Apple Vision Pro discussion

Brad Lynch uses his Apple Vision Pro daily and has even replaced his monitor with the AVP. He regularly uses the “Personas” (Avatars) when talking with co-workers and others in the VR community. But now refrains from using the Personas when talking with others “out of respect.” I have only used it very occasionally since doing my initial evaluation for this blog.

29:10 Mixed Reality while driving (is a bad idea)

Jeri brings up the “influencers” that bought (and likely returned in the two-week return window) and Apple Vision Pro may a viral YouTube video driving around in a Cyber Truck. We then discuss how driving around this was is dangerous.

Next Video – Meta Orion

In the next video in this series, we discuss Meta Orion.

Update (Oct. 19th, 2024)

While the general premise of this article is that Meta Orion is using similar waveguide technology to Snap (Wave Optics) and that Magic Leap 2 is correct, it turns out that a number of assumptions about the specifics of what the various companies actually used in their products were incorrect. One of my readers (who wishes to remain anonymous) with deep knowledge of waveguides responded to my request for more information on the various waveguides. This person had both a theoretical knowledge of waveguides and what Meta Orion, Wave Optics (now Snap), Magic Leap Two, and Hololen 2 used.

My main error about the nature of waveguide “grating” structures was a bias toward linear gratings, with which I was more familiar. I overlooked the possibility that Wave Optics was using a set of “pillar” gratings that act like a 2D set of linear gratings.

A summary of the corrections:

1. Hololens 2 had a two-sided waveguide. The left and right expansion gratings are on opposite sides of the waveguide.
2. Prior Wave Optics (Snap) waveguides use a pillar-type 2-D diffraction grating on one side. There is a single waveguide for full color. The new Snap Spectacles 5 is likely (not 100% sure) using linear diffraction gratings on both sides of a single waveguide full color, as shown in this article.
3. Magic Leap Two uses linear diffraction gratings on both sides of the waveguide. It does use three waveguides.

The above corrections indicate that Meta Orion, Snap Spectacles 5 (Wave Optics), and Magic Leap all have overlapping linear gratings on both sides. Meta Orion and Snap likely use a single waveguide for full color, whereas the Magic Leap 2 has separate waveguides for the three primary colors.

I’m working on an article that will go into more detail and should appear soon, but I wanted to get this update out quickly.

Introduction and Background

After my last article, Meta Orion AR Glasses (Pt. 1 Waveguides), I got to thinking that the only other diffractive grating waveguide I have seen with a 2-D (X-Y) expansion and exit gratings, used in Meta’s Orion, was from Wave Optics (purchased by Snap in May 2021)

The unique look of Wave Optics waveguides is how I easily identified that Snap was using them before it was announced that Snap had bought Wave Optics in 2021 (see Exclusive: Snap Spectacles Appears to Be Using WaveOptics and [an LCOS] a DLP Display).

I then wondered what Magic Leap Two (ML2) did to achieve its 70-degree FOV and uncovered some more interesting information about Meta’s Orion. The more I researched ML2, the more similarities I found with Meta’s Orion. What started as a short observation that Meta Orion’s waveguide appears to share commonality with Snap (Wave Optics) waveguides ballooned up when I discovered/rediscovered the ML2 information.

Included in this article is some “background” information from prior articles to help compare and contrast what has been done before with what Meta’s Orion, Snap/Wave Optics, and Magie Leap Two are doing.

Diffractive Waveguide Background

I hadn’t looked at in any detail how Wave Optics diffraction gratings worked differently before. All other diffraction (I don’t know about holographic) grating waveguides I had seen before used three (or four) separate gratings on the same surface of the glass. There was an Entrance Grating, a first expansion and turning grating, and then a second expansion and exit grating. The location and whether the first expansion grating was horizontal or vertical varied with different waveguides.

Hololens 2 had a variation with left and right horizontal expansion and turning gratings and a single exit grating to increase the field of view. Still, all the gratings were on the same side of the waveguide.

Diffraction gratings bend light based on wavelength, similar to a prism. But unlike a prism, a grating will bend the light in a series of “orders.” With a diffractive waveguide, only the light from one of these orders is used, and the rest of the light is not only wasted but can cause problems, including “eye glow” and reduce the contrast of the overall system

Because diffraction is wavelength-based, it bends different colors/wavelengths in different amounts. This causes issues when sending more than one color through a single waveguide/diffraction grating. These problems are compounded as the size of the exit grating and FOV increases. Several diffraction waveguide companies have one (full color), or two (red+blue and blue+green) waveguides for smaller FOVs and then use three waveguides for wider FOVs.

For more information, Quick Background on Diffraction Waveguides, MicroLEDs and Waveguides: Millions of Nits-In to Thousands of Nits-Out with Waveguides, and Magic Leap, HoloLens, and Lumus Resolution “Shootout” (ML1 review part 3).

Meta Orion’s and Wave Optics Waveguides

I want to start with a quick summary of Orion’s waveguide, as the information and figures will be helpful in comparing it to that of Wave Optics (owned by Snap and in Snap’s Spectacles AR Glasses) and the ML2.

Summary of Orion’s waveguide from the last article

Orion’s waveguide appears to be using a waveguide substrate with one entrance grating per primary color and then two expansion and exit/output gratings. The two (crossed) output gratings are on opposite sides of the Silicon Carbide (SiC) substrate, whereas most diffractive waveguides use glass, and all the gratings are on one side.

Another interesting feature shown in the patents and discussed by Meta CTO Bosworth in some of his video interviews about Orion is “Disparity Correction,” which has an extra grating used by other optics and circuitry to detect if the waveguides are misaligned. This feature is not supported in Orion, but Bosworth says it will be included in future iterations that will move the input grating to the “eye side” of the waveguide. As shown in the figure below, and apparently in Orion, light enters the waveguide from the opposite side of the eyes. Since the projectors are on the eye side (in the temples), they require some extra optics, which, according to Bosworth, make the Orion frames thicker.

Wave Optics (Snap) Dual-Sided 2D Expanding Waveguide

Wave Optics US patent application 2018/0210205 is based on the first Wave Optics patent from the international application WO/2016/020643, first filed in 2014. FIG 3 (below) shows a 3-D representation of diffraction grating with an input grating (H0) and cross gratings (H1 and H2) on opposite sides of a single waveguide substrate.

The patent also shows that the cross gratings (H1 and H2) are on opposite sides of a single waveguide (FIG. 15B above) or one side of two waveguides (FIG. 15A above). I don’t know if Wave Optics (Snap) uses single- or double-sided waveguides in its current designs, but I would suspect it is double-sided.

While on the subject of Wave Optics waveguide design, I happen to have a picture of a Wave Optics 300mm glass wafer with 24 waveguides (right). I took the picture in the Schott booth at AR/VR/MR 2020. In the inset, I added Meta’s picture of the Orion 100mm SiC wafer, roughly to scale, with just four waveguides.

By the way, in my May 2021 article Exclusive: Snap Spectacles Appears to Be Using WaveOptics and [an LCOS] a DLP Display, I assumed that Spectacles would be using LCOS in 2021 since WaveOptics was in the process of moving to LCOS when they were acquired. I was a bit premature, as it took until 2024 for Spectacles to use LCOS.

In my hurry in putting together information and digging for connection, it was looking to me that WaveOptics would be using an LCOS microdisplay. As I pointed out, WaveOptics had been moving away from DLP to LCOS with their newer designs. Subsequent information suggests that WaveOptics was still using their much older DLP design. It is still likely that future versions will use LCOS, but the current version apparently does not.

Magic Leap

Magic Leap One (ML1) “Typical” Three Grating Waveguide

This blog’s first significant article about Magic Leap was in November 2016 (Magic Leap: “A Riddle Wrapped in an Enigma”). Since then, Magic Leap has been discussed in about 90 articles. Most other waveguide companies coaxially input all colors from a single projector. However, even though the ML1 had a single field sequential color LCOS device and projector, the LED illumination sources are spatially arranged so that the image from each color output is sent to a separate input grating. ML1 had six waveguides, three for each of the two focus planes, resulting in 6 LEDs (two sets of R, G, & B) and six entrance gratings (see: Magic Leap House of Cards – FSD, Waveguides, and Focus Planes).

Below is a diagram that iFixit developed jointly with this blog. It shows a side view of the ML1 optical path. The inset picture in the lower right shows the six entrance gratings of the six stacked waveguides.

Below left is a picture of the (stack of six) ML1 waveguides showing the six entrance gratings, the large expansion and turning gratings, and the exit gratings. Other than having spatially separate entrance gratings, the general design of the waveguides is the same as most other diffractive gratings, including the Hololens 1 shown in the introduction. The expansion gratings are mostly hidden in the ML1’s upper body (below right). The large expansion and turning grating can be seen as a major problem in fitting a “typical” diffractive waveguide into an eyeglass form factor, which is what drove Meta to find an alternative that goes beyond the ML1’s 50-degree FOV.

Figure 18 from US application 2018/0052276 diagrams the ML1’s construction. This diagram is very close to the ML1’s construction down to the shape of the waveguide and even the various diffraction grating shapes.

Magic Leap Two (ML2)

The ML1 failed so badly that very few were interested in the ML2 compared to the ML1. There is much less public information about the second-generation device, and I didn’t buy an ML2 for testing. I have covered many of the technical aspects of ML2, but I haven’t studied the waveguide before. With the ML2 having a 70-degree FOV compared to the ML1’s 50-degree FOV, I became curious about how they got it to fit.

To start with, the ML2 eliminated the ML1’s support for two focus planes. This cut the waveguides in half and meant that the exit grating of the waveguide didn’t need to change the focus of the virtual image (for more on this subject, see: Single Waveguide Set with Front and Back “Lens Assemblies”).

Looking through the Magic Leap patent applications, I turned up US 2018/0052276 to Magic Leap, which shows a 2-D combined exit grating. US 2018/0052276 is what is commonly referred to in the patent field as an “omnibus patent application,” which combines a massive number of concepts (the application has 272 pages) in a single application. The application starts with concepts in the ML1 (including the just prior FIG 18) and goes on to concepts in the ML2.

This application, loosely speaking, shows how to take the Wave Optics concept of two crossed diffraction gratings on different sides of a waveguide and integrate them onto the same side of the waveguide.

Magic Leap patent application 2020/0158942 describes in detail how the two crossed output gratings are made. It shows the “prior art” (Wave Optics and Meta Orion-like) method of two gratings on opposite sides of a waveguide in FIG. 1 (below). The application then shows how the two crossed gratings can be integrated into a single grating structure. The patent even includes scanning electron microscope photos of the structures Magic Leap had made (ex., FIG 5), which demonstrates that Magic Leap had gone far beyond the concept stage by the time of the application’s filing in Nov. 2018.

I then went back to pictures I took of Magic Leap’s 2022 AR/VR/MR conference presentation (see also Magic Leap 2 at SPIE AR/VR/MR 2022) on the ML2. I realized that the concept of a 2D OPE+EPE (crossed diffraction gratings) was hiding in plain sight as part of another figure, thus confirming that ML2 was using the concept. The main topic of this figure is “Online display calibration,” which appears to be the same concept as Orion’s “disparity correction” shown earlier.

The next issue is whether the ML2 used a single input grating for all colors and whether it used more than one waveguide. It turns out that these are both answered in another figure from Magic Leap’s 2022 AR/VR/MR presentation shown below. Magic Leap developed a very compact projector engine that illuminates and LCOS panel through the (clear) part of the waveguides. Like the ML1, the red, green, and blue illumination LEDs are spatially separated, which, in turn, causes the light out of the projector lens to be spatially separated. There are then three spatially separate input gratings on three waveguides, as shown.

Based on the ML2’s three waveguides, I assumed it was too difficult or impossible to support the “crossed” diffraction grating effect while supporting full color in a single wide FOV waveguide.

Summary: Orion, ML2, & Wave Optics Waveguide Concepts

Orion, ML2, and Wave Optics have some form of two-dimensional pupil expansion using overlapping diffraction gratings. By overlapping gratings, they reduce the size of the waveguide considerably over the more conventional approach, with three diffraction gratings spatially separate on a single surface.

To summarize:

• Meta Orion – “Crossed” diffraction gratings on both sides of a single SiC waveguide for full color.
• Snap/Wave Optics – “Crossed” diffraction gratings on both sides of a single glass waveguide for full color. Alternatively, “crossed” diffraction waveguides on two glass waveguides for full color (I just put a request into Snap to try and clarify).
• Magic Leap Two – A single diffraction grating that acts like a crossed diffraction grating on high index (~2.0) glass with three waveguides (one per primary color).

The above is based on the currently available public information. If you have additional information or analysis, please share it in the comments, or if you don’t want to share it publicly, you can send a private email to newsinfo@kgontech.com. To be clear, I don’t want stolen information or any violation of NDAs, but I am sure there are waveguide experts who know more about this subject.

What about Meta Orion’s Image Quality?

I have not had the opportunity to look through Meta’s Orion or Snap Spectacles 5 and have only seen ML2 in a canned demo. Unfortunately, I was not invited to demo Meta’s Orion, no less have access to one for evaluation (if you can help me gain (legal) access, contact me at newsinfo@kgontech.com).

I have tried the ML2 a few times. However, I have never had the opportunity to take pictures through the optics or use my test patterns. From my limited experience with the ML2, it is much better in terms of image quality than the ML1 (which was abysmal – see Magic Leap Review Part 1 – The Terrible View Through Diffraction Gratings), it still has significant issues with color uniformity like other wide (>40-degree) FOV diffractive waveguides. If someone has a ML2 that I can borrow for evaluation, please get in touch with me at newsinfo@kgontech.com.

I have been following Wave Optics (now Snap) for many years and have a 2020-era Titan DLP-based 40-degree FOV Wave Optics evaluation unit (through the optics picture below). Wave Optics Titan, I would consider a “middle of the pack” (I had seen better and worse) diffractive waveguide at that time. I have seen what seem to be better diffractive waveguides before and since, but it is hard to compare them objectively as they have different FOVs, and I was not able to use my content but rather curated demo content. Wave Optics seemed to be showing better waveguides at shows before being acquired by Snap 2021, but once again, that was with their demo content with short views at shows. I am working on getting a Spectacles 5 to do a more in-depth evaluation and see how it has improved.

Without the ability to test, compare, and contrast, I can only speculate about Meta Orion’s image quality based on my experience with diffractive waveguides. The higher index of refraction of SiC helps as there are fewer TIR bounces, which degrades image quality, but it is far from a volume production-ready technology. I’m concerned about image uniformity with a large FOV and even more so with a single set of diffraction gratings as diffraction is based on wavelength (color).

Lumus Reflective Waveguide Rumors

In Meta Orion AR Glasses: The first DEEP DIVE into the optical architecture, it stated:

There were rumors before that Meta would launch new glasses with a 2D reflective (array) waveguide optical solution and LCoS optical engine in 2024-2025. With the announcement of Orion, I personally think this possibility has not disappeared and still exists.

The “reflective waveguide” would most likely be a reference to Lumus’s reflective waveguides. I have seen a few “Lumus clone” reflective waveguides from Chinese companies, but their image quality is very poor compared to Lumus. In the comment section of my last article, Ding, on October 8, 2024, wrote:

There’s indeed rumor that Meta is planning an actual product in 2025 based on LCOS and Lumus waveguide. 

Lumus has demonstrated impressive image quality in a glasses-like form factor (see my 2021 article: Exclusive: Lumus Maximus 2K x 2K Per Eye, >3000 Nits, 50° FOV with Through-the-Optics Pictures). Since the 2021 Maximus, they have been shrinking the form factor and improving support for prescription lens integration with their new “Z-lens” technology. Lumus claims its Z-Lens technology should be able to support greater than a 70-degree FoV in glass. Lumus also says because their waveguides support a larger input pupil, they should have a 5x to 10x efficiency advantage.

The market question about Lumus is whether they can make their waveguide cost-effectively in mass production. In the past, I have asked their manufacturing partner, Schott, who says they can make it, but I have yet to see a consumer product around the Z-Lens. It would be interesting to see if a company like Meta had put the kind of money they invested into complex Silicon Carbide waveguides into reflective waveguides.

While diffractive waveguides are not inexpensive, they are considered less expensive at present (except, of course, for Meta Orion’s SiC waveguides). Perhaps an attractive proposition to researchers and propriety companies is that diffraction waveguides can be customized more easily (at least on glass).

Not Addressing Who Invented What First

I want to be clear: this article does not in any way make assertions about who invented what first or whether anyone is infringing on anyone else’s invention. Making that determination would require a massive amount of work, lawyers, and the courts. The reason I cite patents and patent applications is that they are public records that are easily searched and often document technical details that are missing from published presentations and articles.

Conclusions

There seems to be a surprising amount of commonality between Meta’s Orion, the Snap/Wave Optics, and the Magic Leap Two waveguides. They all avoided the “conventional” three diffraction gratings on one side of a waveguide to support a wider FOV in an eyeglass form factor. Rediscovering that the ML2 supported “dispersion correction,” as Meta refers to it, was a bit of a bonus.

As I wrote last time, Meta’s Orion seems like a strange mix of technology to make a big deal about at Meta Connect. They combined a ridiculously expensive waveguide with a very low-resolution display. The two-sided diffraction grating Silicon Carbide waveguides seem to be more than a decade away from practical volume production. It’s not clear to me that even if they could be made cost-effective, they would have as good a view out and the image quality of reflective waveguides, particularly at wider FOVs.

Meta could have put together a headset with technology that was within three years of being ready for production. As it is, it seemed like more of a stunt in response to the Apple Vision Pro. In that regard, the stunt seems to have worked in the sense that some reviewers were reminded of seeing the real world directly with optical AR/MR beats, looking at it through camera and display.

Introduction

While Meta’s announced Orion prototype AR Glasses at Meta Connect made big news, there were few technical details beyond it having a 70-degree field of view (FOV) and using Silicon Carbide waveguides. While they demoed to the more general technical press and “influencers,” they didn’t seem to invite the more AR and VR-centric people who might be more analytical. Via some Meta patents, a Reddit post, and studying videos and articles, I was able to tease out some information.

This first article will concentrate on Orion’s Silicon Carbide diffractive waveguide. I have a lot of other thoughts on the mismatch of features and human factors that I will discuss in upcoming articles.

Wild Enthusiasm Stage and Lack of Technical Reviews

In the words of Yogi Berra, “It’s like deja vu all over again.” We went through this with the Apple Vision Pro, which went from being the second coming of the smartphone to almost disappearing earlier this year. This time, a more limited group of media people has been given access. There is virtually no critical analysis of the display’s image quality or the effect on the real world. I may be skeptical, but I have seen dozens of different diffractive waveguide designs, and there must be some issues, yet nothing has been reported. I expect there are problems with color uniformity and diffraction artifacts, but nothing was mentioned in any article or video. Heck, I have yet to see anyone mention the obvious eye glow problem (more on this in a bit).

The Vergecast podcast video discusses some of the utility issues and their related video, Exclusive: We tried Meta’s AR glasses with Mark Zuckerberg, which gives some more information about the experience. Thankfully, unlike Meta or any other (simulated) through-the-optics videos, The Verge clearly marked the videos as “Simulated” (screen capture on the right).

As far as I can tell, there are no true “through-the-optics” videos or pictures (likely at Meta’s request). All the images and videos I found that may look like they could have been taken through the optics have been “simulated.”

Another informative video was by Norm Chan of Adam Savages Tested, particularly in the last two-thirds of the video after his interview with Meta CTO Andrew Bosworth. Norm discussed that the demo was “on rails” with limited demos in a controlled room environment. I’m going to quote Bosworth a few times in this article because he added information; while he may have been giving some level of marketing spin, he seems to be generally truthful, unlike former Hololens 2 leader Alex Kipman, who was repeatedly dishonest in his Hololens 2 presentation (which I documented in several articles including Hololens 2 and why the resolution math fails, and Alex Kipman Fibbing about the field of view, Alex Kipman’s problems at Microsoft with references to other places where Kipman was “fibbing,” and Hololens 2 Display Evaluation (Part 2: Comparison to Hololens 1) or input “Kipman” on this blog’s search feature)

I’m not against companies making technology demos in general. However, making a big deal about a “prototype” and not a “product” at Meta Connect rather than at a technical conference like Siggraph indicates AR’s importance to Meta. It invites comparisons to the Apple Vision Pro, which Meta probably intended.

It is a little disappointing that they also only share the demos with selected “invited media” that, for the most part, lack deep expertise in display technology and are easily manipulated by a “good” demo (see Appendix: “Escape from a Lab” and “Demos Are a Magic Show”). They will naturally tend to pull punches to keep access to new product announcements from Meta and other major companies. As a result, there is no information about the image quality of the virtual display or any reported issues looking through the waveguides (which there must be).

Eye Glow

I’ve watched hours of videos and read multiple articles, and I have yet to hear anyone mention the obvious issue of “eye glow” (front projection). They will talk about the social acceptance of them looking like glasses and being able to see the person’s eyes, but then they won’t mention the glaring problem of the person’s eyes glowing. It stuck out to me because they didn’t mention the eye glow issue, evident in all the videos and many photos.

Eye glow is an issue that diffractive waveguide designers have been trying to reduce/eliminate for years. Then there are Lumus reflective waveguides with inherently little eye glow. Vuzix, Digilens, and Dispelix make big points about how they have reduced the problem with diffractive waveguides (see Front Projection (“Eye Glow”) and Pantoscoptic Tilt to Eliminate “Eye Glow”). However, these diffractive waveguide designs with greatly reduced eye glow issues have relatively small (25-35 degree) FOVs. The Orion design supports a very wide 70-degree FOV while trying to make it fit the size of a “typical” (if bulky) glasses frame; I suspect that the design methods to meet the size and FOV requirements meant that the issue of “eye glow” could not be addressed.

Light Transmission (Dimming?)

The transmissivity seems to vary in the many images and videos of people wearing Orions. It’s hard to tell, but it seems to change. On the right, two frames switch back and forth, and the glasses darken as the person puts them on (from video Orion AR Glasses: Apple’s Last Days)

Because I’m judging from videos and pictures with uncontrolled lighting, it’s impossible to know the transmissivity, but I can compare it to other AR glasses. Below are the highly transmissive Lumus Maximus glasses with greater than 80% transmissivity and the Hololens 2 with ~40% compared to the two dimming levels of the Orion glasses.

Below is a still frame from a Meta video showing some of the individual parts of the Orion glasses. They appear to show unusually dark cover glass, a dimming shutter (possibly liquid crystal) with a drive circuit attached, and a stack of flat optics with the waveguide with electronics connected to it. In his video, Norm Chen stated, “My understanding is the frontmost layer can be like a polarized layer.” This seems consistent with what appears to be the cover “glass” (which could be plastic), which looks so dark compared to the dimming shutter (LC is nearly transparent as it only changes the polarization of light).

If it does use a polarization-based dimming structure, this will cause problems when viewing polarization-based displays (such as LCD-based computer monitors and smartphones).

Orion’s Unusual Diffractive Waveguides

Axel Wong‘s analysis of Meta Orion’s Waveguide, which was translated and published on Reddit as Meta Orion AR Glasses: The first DEEP DIVE into the optical architecture, served as a starting point for my study of the Meta Orions optics, and I largely agree with his findings. Based on the figures he showed, his analysis was based on Meta Platforms’ (a patent holding company of Meta) US patent application 2024/0179284. Three figures from that application are shown below.

[10-08-2024 – Corrected the order of the Red, Green, and Blue inputs in Fig 10 below]

Overlapping Diffraction Gratings

It appears that Orion uses waveguides with diffraction gratings on both sides of the substrate (see FIG. 12A above). In Figure 10, the first and second “output gratings” overlap, which suggests that these gratings are on different surfaces. Based on FIGs 12A and 7C above, the gratings are on opposite sides of the same substrate. I have not seen this before with other waveguides and suspect it is a complicated/expensive process.

As Alex Wong pointed out in his analysis, supporting such a wide FOV in a glass form factor necessitated that the two large gratings overlap. Below (upper-left) is shown the Hololens 1 waveguide, typical of most other diffractive waveguides. It consists of a small input grating, a (often) trapezoidal-shaped expansion grating, and a more rectangular second expansion and output/exit grating. In the Orion (upper right), the two larger gratings effectively overlap so that the waveguide fits in the eyeglasses form factor. I have roughly positioned the Hololens 1 and Orion waveguides at the same vertical location relative to the eye.

Also shown in the figure above (lower left) is Orion’s waveguide wafer, which I used to generate the outlines of the gratings, and a picture (lower right) showing the two diffraction gratings in the eye glow from Orion.

It should be noted that while the Hololens 1 has only about half the FOV of the Orion, the size of the exit gratings is similar. The size of the Hololens 1 exit grating is due to the Hololen 1 having enough eye relief to support most wearing glasses. The farther away the eye is from the grating, the bigger the grating needs to be for a given FOV.

Light Entering From the “wrong side” of the waveguide

The patent application figures 12A and 7C are curious because the projector is on the opposite side of the waveguide from the eye/output. This would suggest that the projectors are outside the glasses rather than hidden in the temples on the same side of the waveguide as the eye.

Meta’s Bosworth in The WILDEST Tech I’ve Ever Tried – Meta Orion at 9:55 stated, “And so, this stack right here [pointing to the corner of the glasses of the clear plastic prototype] gets much thinner, actually, about half as thick. ‘Cause the protector comes in from the back at that point.”

Based on Bosworth’s statement, some optics route the light from the projectors in the temples to the front of the waveguides, necessitating thicker frames. Bosworth said that the next generation’s waveguides will accept light from the rear side of the waveguide. I assume that making the waveguides work this way is more difficult, or they would have already done it rather than having thicker frames on Orion.

However, Bosworth said, “There’s no bubbles. Like you throw this thing in a fish tank, you’re not gonna see anything.” This implies that everything is densely packed into the glasses, so other than saving the volume of the extra optics, there may not be a major size reduction possible. (Bosworth referenced Steve Jobs Dropping an iPod prototype in water story to prove that it could be made smaller due to the air bubbles that escaped)

Disparity Correction (Shown in Patent Application but not in Orion)

Meta’s application 2024/0179284, while showing many other details of the waveguide, is directed to “disparity correction.” Bosworth discusses in several interviews (including here) that Orion does not have disparity correction but that they intend to put it in future designs. As Bosworth describes it, the disparity correction is intended to correct for any flexing of the frames (or other alignment issues) that would cause the waveguides (and their images relative to the eyes) to move. He seems to suggest that this would allow Meta to use frames that would be thinner and that might have some flex to them.

Half Circular Entrance Gratings

Wong, in the Reddit article, also noticed that small input/entrance gratings visible on the wafer looked to be cut-off circles and commented:

However, if the coupling grating is indeed half-moon shaped, the light spot output by the light engine is also likely to be this shape. I personally guess that this design is mainly to reduce a common problem with SRG at the coupling point, that is, the secondary diffraction of the coupled light by the coupling grating.

Before the light spot of the light engine embarks on the great journey of total reflection and then entering the human eye after entering the coupling grating, a considerable part of the light will unfortunately be diffracted directly out by hitting the coupling grating again. This part of the light will cause a great energy loss, and it is also possible to hit the glass surface of the screen and then return to the grating to form ghost images.

Single Waveguide for all three colors?

The patent application seems to suggest that there is a single (double-sided) waveguide for all three colors (red, green, and blue). Most larger FOV full-color diffractive AR glasses will stack three (red, green, and blue—Examples Hololens One and Magic Leap 1&2) or two waveguides (red+blue and blue+green—Example Hololens 2). Dispelix has single-layer, full-color diffractive waveguides that go up to 50 degrees FOV.

Diffraction gratings have a line spacing based on the wavelengths of light they are meant to diffract. Supporting full color with such a wide FOV in a single waveguide would typically cause issues with image quality, including light fall-off in some colors and contrast losses. Unfortunately, there are no “through the optics” pictures or even subjective evaluations by an independent expert as to the image quality of Orion.

Silicon Carbide Waveguide Substrate

The idea of using silicon carbide for Waveguides it not unique to Meta. Below is an image from GETTING THE BIG PICTURE IN AR/VR, which discusses the advantages of using high-index materials like Lithium Niobate and Silicon Carbide to make waveguides. It is well known that going to a higher index of refraction substrates supports wider FOVs, as shown in the figure below. The problem, as Bosworth points out, is that growing silicon carbide wafers are very expensive. The wafers are also much smaller, enabling fewer waveguides per wafer. From the pictures of Meta’s wafers, they only get four waveguides per wafer, whereas there can be a dozen or more diffractive waveguides made on larger and much less expensive glass wafers.

Bosworth says “Nearly Artifact Free” and with Low “Rainbow” capture

A common issue with diffractive waveguides is that the diffraction gratings will capture light in the real world and then spread it out by wavelength like a prism, which creates a rainbow-like effect.

In Adam Savage’s Tested interview (@~5:10), Bosworth said, “The waveguide itself is nano etched into silicon carbide, which is a novel material with a super high index of refraction, which allows us to minimize the Lost photons and minimize the number of photons we capture from the world, so it minimizes things like ghosting and Haze and rainbow all these artifacts while giving you that field of view that you want. Well it’s not artifact free, it’s very close to artifact-free.” I appreciate that while Bosworth tried to give the advantages of their waveguide technology, he immediately corrected himself when he had overstated his case (unlike Hololens’ Kipman as cited in the Introduction). I would feel even better if they let some independent experts study it and give their opinions.

What Bosworth says about rainbows and other diffractive artifacts may be true, but I would like to see it evaluated by independent experts. Norm said in the same video, “It was a very on-rails demo with many guard rails. They walked me through this very evenly diffused lit room, so no bright lights.” I appreciate that Norm recognized he was getting at least a bit of a “magic show” demo (see appendix).

Wild Enthusiasm Stage and Lack of Technical Reviews

In the words of Yogi Berra, “It’s like deja vu all over again.” We went through this with the Apple Vision Pro, which went from being the second coming of the smartphone to almost disappearing earlier this year. This time, a more limited group of media people has been given access. There is virtually no critical analysis of the display’s image quality or the effect on the real world. I may be skeptical, but I have seen dozens of different diffractive waveguide designs, and there must be some issues, yet nothing has been reported. I’m expecting there to be problems with color uniformity and diffraction artifacts, but nothing was mentioned.

Strange Mix of a Wide FOV and Low Resolution

There was also little to no discussion in the reviews of Orion’s very low angular resolution of only 13 pixels per degree (PPD) spread over a 70-degree FOV (a topic for my next article on Orion). This works to about a 720- by 540-pixel display resolution.

Several people reported seeing a 26PPD demo, but it was unclear if this was a form factor or a lab-bench demo. Even 26PPD is a fairly low angular resolution.

Optical versus Passthough AR – Orion vs Vision Pro

Meta’s Orion demonstration is a declaration that optical AR (e.g., Orion) and non-camera passthrough AR, such as Apple Vision Pro, are the long-term prize devices. It makes the point that no passthrough camera and display combination can come close to competing with the real-world view in terms of dynamic range, resolution, biocular stereo, and infinite numbers of focus depths.

As I have repeatedly pointed out in writing and presentations, optical AR prioritizes the view of the real world, while camera passthrough AR prioritizes the virtual image view. I think there is very little overlap in their applications. I can’t imagine anyone allowing someone out on a factor floor or onto the streets of a city in a future Apple Vision Pro type device, but one could imagine it with something like the Meta Orion. And I think this is the point that Meta wanted to make.

Conclusions

I understand that Meta was demonstrating, in a way, “If money was not an obstacle, what could we do?” I think they were too fixated on the very wide FOV issue. I am concerned that the diffractive Silicon Carbide waveguides are not the right solution in the near or long term. They certainly can’t have a volume/consumer product with a significant “eye glow” problem.

This is a subject I have discussed many times, including in Small FOV Optical AR Discussion with Thad Starner and FOV Obsession. They have the worst of all worlds in some ways, with a very large FOV and a relatively low-resolution display; they block most of the real world for a given amount of content. With the same money, I think they could have made a more impressive demo with exotic waveguide materials that didn’t seem so far off in the future. I intend to get more into the human factors and display utility in this series on Meta Orion.

Appendix: “Demos Are a Magic Show”

Seeing the way Meta introduced Orion and hearing of the crafted demos they gave reminded me of one of my earliest blog articles from 2012 call Cynics Guide to CES – Glossary of Terms which gave warning about seeing demos.

Escaped From the Lab

Orion seems to fit the definition of an “escape from the lab.” Quoting from the 2012 article:

“Escaped from the lab” – This is the demonstration of a product concept that is highly impractical for any of a number of reasons including cost, lifetime/reliability, size, unrealistic setting (for example requires a special room that few could afford), and dangerous without skilled supervision.  Sometimes demos “escape from the lab” because a company’s management has sunk a lot of money into a project and a public demo is an attempt to prove to management that the concepts will at least one day appeal to consumers.

I have used this phrase a few times over the years, including The Hololens 2 (Hololens 2 Video with Microvision “Easter Egg” Plus Some Hololens and Magic Leap Rumors), which was officially discontinued this month, although it has long since been seen as a failed product. I also commented (in Magic Leap Review Part 1 – The Terrible View Through Diffraction Gratings – see my Sept. 27, 2019 comment) that the Magic Leap One was “even more of a lab project.”

Why make such a big deal about Orion, a prototype with a strange mix of features and impractically expensive components? Someone(s) is trying to prove that the product concept was worth continued investment.

Magic Show

I also warned that demos are “a magic show.”

A Wizard of Oz (visual) – Carefully controlling the lighting, image size, viewing location and/or visual content in order to hide what would be obvious defects.   Sometimes you are seeing a “magic show” that has little relationship to real world use.

I went into further detail in this subject in my early coverages of the Hololens 2 in the section, “Demos are a Magic Show and why are there no other reports of problems?“:

I constantly try and remind people that “demos are a magic show.” Most people get wowed by the show or being one of the special people to try on a new device. Many in the media may be great at writing, but they are not experts on evaluating displays. The imperfections and problems go unnoticed in a well-crafted demo with someone that is not trained to “look behind the curtain.”

The demo content is often picked to best show off a device and avoid content that might show flaws. For example, content that is busy with lots of visual “noise” will hide problems like image uniformity and dead pixels. Usually, the toughest test patterns are the simplest, as one will immediately be able to tell if something is wrong. I typically like patterns with a mostly white screen to check for uniformity and a mostly black screen to check for contrast, with some details in the patterns to show resolution and some large spots to check for unwanted reflections. For example, see my test patterns, which are free to download. When trying on a headset that supports a web browser, I will navigate to my test pattern page and select one of the test patterns.

Most of the companies that are getting early devices will have a special relationship with the manufacturer. They have a vested interest in seeing that the product succeeds either for their internal program or because they hope to develop software for the device. They certainly won’t want to be seen as causing Microsoft problems. They tend to direct their negative opinions to the manufacturer, not public forums.

Only with independent testing by people with display experience using their own test content will we understand the image quality of the Hololens 2.

Introduction

Sometimes, companies make what seems, on the surface, technically poor decisions. I consider this the case with Canon’s new R5 Mark ii (and R1) dropping support for sensor Pixel Shifting High Resolution (what Canon calls IBIS High Res). Canon removed the IBIS High Res mode, which captures (as I will demonstrate) more real information and seemingly adds an AI upscaling to create fake information. AI upscaling, if desired, can be done better and more conveniently on a computer, but Pixel Shift/IBIS High Res cannot.

The historical reason for pixel shift is to give higher resolution in certain situations. Still, because the cameras combine the images “in-camera” with the camera’s limited processing and memory resources plus simple firmware algorithms, they can’t deal with either camera or subject motion. Additionally, while the Canon R5 can take 20 frames per second (the R5 Mark ii can take 30 frames per second), taking the nine frames takes about half a second, but then it takes another ~8 seconds for the camera to process them. Rather than putting more restrictions on shooting, it would have been much easier and faster to save the raw frames (with original sensor subpixels) to the flash drive for processing later by a much more capable computer using better algorithms that can constantly be improved.

Canon’s competitors, Sony and Nikon, are already saving raw files with their pixel-shift modes. I hoped Canon would see the light with the new R5 mark ii (R5m2) and support IBIS HR in saving the raw frames. Instead, Canon went in the wrong direction; they dropped IBIS High Res altogether and added an in-camera “AI upscaling.” computer. The first-generation R5 didn’t have IBIS High Res, but a firmware release later added this capability. I’m hoping the same will happen with the R5 Mark ii, only this time saving the RAW frames rather than creating an in-camera JPEG.

Features Versus Capabilities

I want to distinguish between a “feature” and a “capability.” Take, for example, high dynamic range. The classical photography problem is taking a picture in a room with a window with a view; you can expose inside the room, in which case the view out the window will be blown out, or you expose the view out the window, in which case the room will look nearly black. The Canon R5 has an “HDR Mode” that takes multiple frames at different exposure settings and allows you to save a single processed image only or with all the frames saved. The “feature” was making a single HDR image, and the “capability” was rapidly taking multiple frames with different exposures and saving those frames.

The Canon R5 made IBIS High Res a feature when it only offered a single JPEG output without the capability of saving individual frames with the sensor shifted by sub-pixel amounts. By saving raw frames, the software could better combine frames. Additionally, the software could deal with camera and subject motion, which are unsavable artifacts in an IBIS high-res JPEG. As such, when I use IBIS High Res, I typically take three pictures just in case, as one of the pictures often will have unfixable problems that can only be seen once viewed on a computer monitor. It would also be desirable to select how many frames to save; for example, saving more than one cycle of frames would help deal with subject or camera motion.

Cameras today support some aspects of “computational photography.” Saving multiple images can be used for panoramic stitching, high dynamic range, focus stacking (to support larger than possible depths of focus with a single picture), and astrophotography image stacking (using interval timers to take many shots that are added together). Many cameras, like the R5, have even added modes to support taking multiple pictures for focus stacking, high dynamic range, and interval timers. So for the R5 mk. ii to have dropped sensor pixel shifting seems like a backward direction in the evolution of photography.

This Blog’s Use of Pixel Shifting for Higher Resolution

Both cameras have “In-Body-Stabilization” (IBIS) that normally moves the camera sensor based on motion detection to reduce camera/lens motion blur. They both also support a high-resolution mode where, instead of using the IBIS for stabilization, they use it to shift the sensor by a fraction of a pixel to take a higher-resolution image. Canon called this capability “IBIS High Res.” The R5 in-camera combines nine images, each shifted by 1/3rd of a pixel, to make a 405mp JPEG image. The D5 combines four images, each shifted by a half pixel.

In the past year, I started using my “personal camera,” the Canon R5 (45MP “full frame” 35mm), to take pictures of VR/Passthrough-AR and optical AR glasses (where possible). I also use my older Olympus D5 Mark iii (20MP Micro 4/3rd) because it is a smaller camera with smaller lenses that lets it get into the optimum optical location in smaller form factor AR glasses.

The cameras and lenses I use most are shown on the right, except for the large RF15-35mm lens on the R5 camera, which is shown for comparison. To take pictures through the optics and get inside the eye box/pupil, the lens has to be physically close to the image sensor in the camera, which limits lens selection. Thus, while the RF15-35mm lens is “better” than the fixed focus 28mm and 16mm lenses, it won’t work to take a headset picture. The RF28mm and RF16mm lenses are the only full-frame Canon lenses I found to work. Cell phones with small lenses “work,” but they don’t have the resolution of a dedicated camera, aperture control, and shutter speed control necessary to get good pictures through headsets.

Moiré

In addition to photography being my hobby, I take tens of thousands of pictures a year via the optics of AR and VR headsets, which pose particular challenges for this blog. Because I’m shooting at displays with a regular pattern of pixels with a camera its regular pattern of pixels, there is a constant chance for moiré due to the beat frequencies between the pixels and color subpixels of the camera and the display device as magnified by the camera and headset optics (left). To keep within the eye box/pupil of the headset, I am limited to simpler lenses that are physically short to keep the distance from the headset optics to the camera short, which limits the focal lengths and thus magnification to combat moiré. In camera, pixel-shifting has proven to be a way to not only improve resolution but greatly reduce moiré effects.

Issues with moiré are not limited to taking pictures via AR and VR headsets; it is a problem with real-world pictures that include things like patterns in clothing (famously with fences (from a distance where they form a small pattern) and other objects with a regular pattern (see typical photographic moiré problems below).

Anti-Aliasing

Those who know signal theory know that a low-pass cutoff filter reduces/avoids aliasing (moiré is a form of aliasing). Cameras have also used “anti-aliasing” filters, which very slightly blur the image to reduce aliasing, but this comes at the expense of resolution. In the past, with lower-resolution sensors, the chance of encountering real-world things in a picture that would cause aliasing was more likely, and the anti-aliasing filters were more necessary.

As the resolution of sensors has increased, there is a lesser likelihood that something in the typical picture that is in focus will be at the point it aliases and combined with better algorithms that can detect and reduce the effect of moiré. Still, while sometimes the moiré can be fixed in post-processing, in critical or difficult situations, it would be better if additional frames were stored to clue software into processing it as aliasing/moiré rather than “real” information.

Camera Pixels and Bayer Filter (and misunderstanding)

Most cameras today (including Canon) use a Bayer Filter pattern (below right) with two green-filtered pixels for each red or blue pixel. When producing an image for a person to view, a computer’s camera or RAW conversion software, often called “debayering” or “demosaicing,” generates a full-color pixel by combining the information from many (8 or more) surrounding single-color pixels with the total number of full-color pixels equaling the number of photosites.

Camera makers count every photosite as a pixel even though the camera only captured “one color” at that photosite. Some people, somewhat mistakenly, think the resolution is one-quarter claimed since only one-quart red and blue photosites exist. After all, with a color monitor, we don’t count the red, green, and blue subpixels as 3 pixels but just one. However, Microsoft’s ClearType does gain some resolution from the color subpixels to refine text better.

It turns out that except for extreme image cases, including special test patterns, the effective camera resolution is close to the number of photosites (and not 1/4th or 1/2). There are several reasons why this is true. First, note the red, green, and blue filter’s frequency responses for the color camera sensor (above left – taken from a Sony sensor as it was available). Notice how their spectrums are wide and overlapping. The wide spectral nature of these filters is necessary to capture all the continuous spectrums of color in the real world (every call “red” does not have the same wavelength). If the filters were very narrow and only captured a single wavelength, then any colors that are not that wavelength would be black. Each photosite captures intensity information for all colors, but the filtering biases it toward bands of colors.

Almost everything (other than spectral lines from plasmas, lasers, and some test patterns) that can be seen in the real world is not a single wavelength but a mix of wavelengths. There is even the unusual case of magenta, which does not have a wavelength (and thus, many claim it is not a color) but is a mix of blue and red. With a typical photo, we have wide-spectrum filters capturing wide-spectrum colors.

It turns out that humans sense resolution mostly in intensity and not color. This fact has been exploited to reduce the bandwidth of early color television and to reduce data in all the video and image compression algorithms. Thanks to the overlap in the color filters in the camera filters, there is considerable intensity information in the various color pixels.

Human Vision and Color

Consider human vision if the camera sensor’s Bayer patterns and color filter spectral overlaps were bad, then consider the human retina. On average, humans have 7 million cones in the retina, of which ~64% are long (L) wavelength (red), ~32% medium (M – green), and ~2% short (S – blue). However, these percentages vary widely from person to person, particularly the percentage of short/blue cones. The cones that sense color support high resolution are concentrated in the center of vision.

Notice the spectral response of the so-called red, green, and blue cones (below left) and compare it to the camera sensor filters’ response above. Note how much the “red” and “green” responses overlap. On the right is a typical distribution of cones near the fovea (center) of vision, and note there are zero “blue”/short cones in the very center of the fovea; it makes the Bayer pattern look great😁.

Acuity of the Eye

Next, we have the fact that the cones are concentrated in the center of vision and that visual acuity falls off rapidly. The charts below show the distribution of rods and cones in the eye (left) and the sharp fall-off in visual acuity from the center of vision.

Saccadic Eye Movement – The Eyes’ “Pixel Shifting”

Looking at the distribution of cones and the lack of visual acuity outside the fovea, you might wonder how humans see anything in detail. The eye constantly moves in a mix of large and small steps known as saccades. The eye tends to blank while it moves and then takes a metaphorical snapshot. The visual cortex takes the saccade’s “snapshots” and forms a composite image. In effect, the human visual system is doing “pixel shifting.”

My Use of Pixel Shifting (IBIS High-Res)

I am a regular user of the IBIS High-Resolution on this blog. Taking pictures of displays with their regular patterns is particularly prone to moiré. Plus, with the limited lenses I can use that are all wide-angle (and thus low magnification), it helps to get some more resolution. With IBIS, a single picture 405 mp (24,576 by 16,384 pixels) IBIS High-Resolution image can capture ~100-degree wide FOV and yet see details of individual pixels from a 4K display device.

It seems a bit afterthought on the R5 with the JPEG output. Even with the camera on a tripod, it screws up, so usually, I take three shots just in case because I will only know later when I look at the results blown up on a monitor if one of them messed up. The close-in crops (right) are from two back-to-back shots with IBIS high-res. In the bad shot, you can see how the edges look feathered/jagged (particularly comparing vertical elements like the “l” in Arial). I would much rather have had the IBIS HR output the 9 RAW images.

IBIS High-Res Comparison to Native Resolution

IBIS High Res helps provide higher resolution and can significantly reduce moiré. Often, the pixel shift output will have much less moiré. I can often reduce the IBIS high-res to a lower resolution, and the image has much less moiré and is a bit sharper even when scaled down to the size of a “native” resolution picture as shown below.

The crops below show the IBIS High Res image at full resolution and the native resolution scaled up to match, along with insets of the IBIS High Res picture scaled down to match the native resolution.

The Image below was taken in IBIS High Resolution and then scaled down by 33.33% for publication on this blog (from the article AWE 2024 VR – Hypervision, Sony XR, Big Screen, Apple, Meta, & LightPolymers).

The crops below compare the IBIS High Res at full resolution to a native image upscaled by 300%. Notice how the IBIS High Res has better color detail. If you look at the white tower on a diagonal in the center of the picture (pointed to by the red arrow), you can see the red (on the left) blue chroma aberrations caused by the headset’s optics, but these and other color details are lost in the native shot.

Conclusions

While my specific needs are a little special, I think Canon is missing out on a wealth of computational photography options by not supporting IBIS High-Res with RAW output. The obvious benefits are helping with moiré and getting higher-resolution still lifes. By storing RAW, there is also the opportunity to deal with movement in the scene, which may even be hand-held. It would be great to have the option to control the shift amount (shift by 1/3 and 1/2 would be good options) and the number of pictures. For example, it would be good to capture more than one “cycle” to help deal with motion.

Smartphones are cleaning up on dedicated cameras in “computational photography” to make small sensors with mediocre optics look very good. Imagine what could be done with better lenses and cameras. Sony, a leader in cell phone sensors, knows this and has pixel shift with RAW output. I don’t understand why Canon is ceding the pixel shift to Sony and Nikon. Hopefully, it will be a firmware update like it was on the original R5. Only this time, please save the RAW/cRAW files.

In related news, I’m working on an article about Texas Instrument’s renewed thrust into AR with DLP. TI DLP has been working with PoLight to support Pixel Shift (link to video with PoLight) for resolution enhancement with AR glasses (see also Cambridge Mechatronics and poLight Optics Micromovement (CES/PW Pt. 6))

Introduction

The recently announced Even Realities G1 (ER G1) glasses (right) enter a somewhat crowded field of glasses form factor AR glasses using Jade Bird Display’s (JBD) 640×480 green (only) MicroLEDs.

While I have seen many of the other glasses with my own eyes, I have only seen the ER G1 glasses in the videos, including a video conference with Even Realities CEO and founder Will Wang where he answered questions for about an hour. The G1 makes an interesting case study regarding the features they supported and, perhaps more importantly, the apparent design trade-offs.

The G1 stands out as the first I have seen to integrate prescription lenses while coming closest to having a form factor that looks like ordinary glasses, with high transparency and “eye glow” control (discussed later). Overall, Even Reality has focused on a more minimalist approach. The glasses cost $599US plus prescriptions (if necessary), which start at $150US for simple far sight-only correction. Even Realities’ prescription glasses can correct for astigmatism, and progressive lenses will be in some regions (at additional cost).

Other JBD MicroLED Glasses

Many companies have shown products using JBD’s MicroLEDs in a glasses-like form factor. They include Vuzix Z100 (monocular), Meta Bounds Mojie, LAWK Metalens, MyVu, INMO Go, and Oppo Air 2. Below are pictures of AR glasses shown in JBD’s SID Display Week 2024 booth.

While I have seen through most of the above with my own eyes, I have only seen the ER G1 glasses in the videos, including a video conference with Even Realities CEO and founder Will Wang where he answered questions for about an hour. The G1 makes an interesting case study regarding the features they supported and, perhaps more importantly, the apparent design trade-offs.

The TCL Ray Neo X2 (3-chip X cube full-color) and Vuzix Shield (biocular, green only), while they use JBD MicroLEDs, are in another category with 8-core CPUs running Android (others have minimal processing just for Bluetooth communication), WiFi, cameras, and other features not found in the other JBD AR glasses. However, this added capability comes at the cost of thicker and heavier frames to support the added feature, including the batteries that drive them. TCL RayNeo X2 and Ray Neo X2 Lite

For some of my discussion of MicroLED AR Glasses, see DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8), TCL RayNeo X2 and Ray Neo X2 Lite, CES (Pt. 2), Sony XR, DigiLens, Vuzix, Solos, Xander, EverySight, Mojie, TCL color µLED, Mixed Reality at CES and the AR/VR/MR 2024 Video (Part 1 – Headset Companies), and AWE 2024 Panel: The Current State and Future Direction of AR Glasses.

ER G1 FOV and Display Location

TT Technology did a YouTube review (and disclosed up front that Even Realities had sponsored it) with multiple videos taken via a smartphone camera looking through the optics. One of these videos gives a rough idea of the image size and location within the FOV (below). The relative size of the virtual image to the glasses frame opening might be a bit off due to the camera not being located in the same place as the eye, but it gives a good idea of how much content can be displayed and the size within the FOV. One thing to notice is that the virtual image is wide and not very tall. As it turns out (later), the ER G1 optics/waveguide only displays about 1/3rd of the height of the JBD 640×480 Microdisplay.

Many, if not most, designers underestimate the importance of not disturbing the view when the display is off for something intended to be “all-day wearable” (the issues for specific task-oriented headsets are different). Designers are focused on providing the benefits of the virtual display without considering the impact. People are not going to wear something all day that hampers their vision. This hampering could be in the form of darkening, reducing sharpness (as can happen when looking through gratings or other combiner optics), or light capture artifacts (ex., “rainbows”). For the G1, this appears to come at the expense of the virtual image size, although it also has some size and weight benefits.

Comparing the ER G1 to the Vuzix Z100 & Ultralite S

The Vuzix Z100 provides an interesting contrast in design decision to the ER G1. At AWE 2024 panel, The Current State and Future Direction of AR Glasses, discussed glass-form-factor (optical) AR glasses that are wirelessly connected to a smartphone. In the Q&A session, Thad Starner, a wearer of AR glasses since 1993, one of the architects of Google Glass, and a Georgia Tech Professor who has long been a proponent of minimalist AR Glasses (see also my 2017 article FOV Obsession), brought up the minimalist issue during the Q&A session (at 49:46 in the video).

At AWE 2024, Thad was wearing the Vuzix Z100 (right), which seems to meet his requirements and is similar in size and functionality with a few notable differences. Both are glass form factors, with the Vuzix having a more “traditional” style frame and the G1 having an arguably more modern look. Both weigh about the same.

At CES 2024, Vuzix demonstrated a new Ultralite S (Sport) design (below right), which was a bit sleeker than the Z100. Vuzix also said they were better at controlling the “eye glow” with this model. I confirmed that the Ultralite S uses a waveguide designed for pantoscopic tilt (to be discussed later), perhaps among other design improvements.

The Vuzix Z100 (and Ultralight S) and the ER G1 use Jade Bird Displays green (only) 640×480 MicroLED displays, diffractive waveguides, and a Bluetooth wireless connection to a smartphone. The ER G1, as discussed, cuts off more than 1/3rd of the vertical pixels. The Z100 and Ultralight S are both monocular.

Biocular Versus Monocular

With his 30+ years of experience, Thad Starner has long stated his preference for a narrower FOV (25 to 30 degrees) monocular display that appears offset to the outside/temple of the user’s straight-ahead view so that notices or the like doesn’t block normal vision. This is the case for the Vuzix Z100 he wore at AWE 2024. Starner was a technical leader of Google Glass, which positioned the display above and on the temple side of the center of view.

Starner argues that the most important thing for an AR display to do is not to block forward vision. In the worst case, it could be dangerous for a notification to block a person’s view at a critical time.

The ER G1 positions the biocular display image above the user’s straight-ahead view so that the virtual images don’t block the critical straight-ahead view. However, in doing so, the waveguide (and optics) cut off most of the vertical pixels from the display device. The G1 does not appear to use all the display’s 640 pixels horizontally for information, which would allow the display to be moved horizontally to adjust for things like IPD.

Having worked with monocular and binocular headsets, I have found that most people will find binocular displays take less time to get used to and thus are more “consumer friendly.” But monocular still works if you can get it on the user’s “dominant eye” (one eye tends to dominate).

Design Tradeoffs

It would be ergonomically better if the virtual images were located below the center of vision rather than above it, as human eye muscles are better at looking down than up. Then, to avoid interference with forward vision, Even Realities has cut off the vertical image height by more than 3X.

However, moving the display/exit grating to the bottom means larger areas of diffraction gratings that would cause light-capturing (“rainbow”) issues. Another issue is that if progressive prescription correction is used, it will put the “far” vision at the top and the “near” vision at the bottom, but the output of waveguides will still appear to be focused far away and thus not in focus if put in the lower area of progressive lenses. Additionally, locating the virtual image below the center of forward vision tends to block when someone is looking downward to read.

I’m not criticizing Even Realities’ decisions but pointing out the many design trade-offs in something as simple as the location of the image. Even Realities made consistent decisions, including accepting the loss of more than one-third of the display’s pixels, which reduced power consumption and heat dissipation, contributing to reducing size, weight, and cost.

Integrated Prescription versus Inserts

The G1s stand out today because they support integrated (optically glued to the waveguide) prescription lenses; all others require snap-in or clip-on prescription inserts to correct vision. This approach is thinner, cleaner-looking, and lighter and eliminates reflection from air gaps between the vision correction lens and the waveguide. But then, if your prescriptions change, you must buy a new set of glasses. Simple prescriptions for nearsighted correction only cost an additional $150US, with astigmatism and progressive costing an unspecified amount more.

The Inserts for the Vuzix Z100 snap into the frames, which gives a more integrated look than those with clip-on corrections. If the prescription changes, only the inserts need to be replaced, or if the smart glasses break, the lenses can be swapped to a working unit. The insert approach would work better in some “enterprise” applications where smart glasses can be shared. However, the snap-in approach will not be as clean-looking as the integrated approach of the G1 and will be more prone to reflections between the waveguide and inserts. The company Lensology makes similar snap-in inserts for TCL’s RanNeo X2.

Pantoscoptic Tilt to Eliminate “Eye Glow”

G1 also uses pantoscopic tilt (right) to direct the diffractive order that causes the “eye glow” downward. In my video conference with Even Realities CEO Will Wang, I could only tell his glasses were turn-on when he tilted his head.

The G1 AR glasses look very clean, and the diffraction gratings seem nearly invisible when looking straight on. The TT Technology YouTube video discussed earlier also had many video segments of the G1 looking at the wearer while the G1 was active. The stills captured from the video below show how the G1 looks from various angles. Looking straight on (left), there is no eye glow, and the waveguide exit grating seems invisible. Looking from the side, and dependent on the lighting, you can see the grating (center). But the eye glow can be seen when the head tilts back far enough (right).

This “trick” is not new. Digilens, Vuzix in the Z100, and Displelix, among others, also use pantoscopic tilt to eliminate eye glow, but many others, including the TLC Rayneo X2, don’t.

Below are some figures from Digitlens and Displelix on the issue and how designing the optics for a tilted waveguide causes the “leakage light/eye glow” to be redirected downward. With diffractive waveguides, for each degree the lenses tilt, the eye glow light is directed downward by two degrees. With this 2X multiplier, it only takes a few degrees of tilt to direct the light to where others won’t normally be able to see it. An observer typically will only see the eye glow if the wearer tilts their head significantly. As Displelix pointed out (below right), other diffractive design factors can also reduce the leakage.

Reduced Rainbow Effect (Diffractive Grating Light Capture)

Even Realities claims that they have designed their waveguides to reduce the diffractive waveguide light capture (commonly referred to as “rainbows” – see right), where the diffraction gratings direct light sources (like overhead lights) into the eye. The same diffractive “physics” that makes waveguides work makes them prone to capturing light at certain angles from the real world and directing it into the eye. Early diffractive waveguides like the Magic Leap One and the first Hololens had major rainbow problems.

Everyone is working on diffraction grating designs and optical coatings to reduce the rainbow issue. Still, without having everyone’s glasses side by side for testing, it is hard to know which is doing the best. The G1 does not appear to have an exposed “expansion grating,” unlike most other waveguides, including the Oppo Air 2 (right), which can contribute to visible rainbow effects.

High Transparency

The Even Reality G1 has very high transparency (better than 90%) and appears more transparent than other monochrome (green) waveguides I have seen. The G1’s waveguides seem nearly invisible in a front-on view (top-right), but as noted earlier, they can be seen by an observer when the glasses are at an angle.

Shown (mid-right) is me at CES 2024, wearing the Vuzix Ultralight S. I’ve added a dotted red line around the waveguide, indicating the boundary of the waveguide’s exit grating (where the image light exits). On the Vuzix Ultralight S, a slight darkening in the area of the exit grating is visible, unlike the G1 in this straight-on view. Also, in the Ultralite S, you can see that at least some of the exit grating is in the forwarded vision, which is good for seeing the virtual display but might cause some diffraction issues when the display is off.

The G1’s exit grating can be seen in the right lighting, such as this view of the G1 in its charger case (bottom-right). Notice that the G1’s exit grating is more than 2x wider horizontally than vertically. The JBD MicroLEDs that both Even Realites, Vuzix (and many other) companies use have 640×480 pixels and thus a 4:3 aspect ratio. The 4:3 aspect ratio can be seen in the Vuzix Ultralite S (above-mid-right) and the Oppo glasses (below, from AR/VR/MR 2023), which use the same JBD green MicroLED.

Also, notice the 4:3 aspect ratio and amount of content in the through-the-optics picture on the right. It was taken handheld with an Olympus D5 Mk. III camera at AR/VR/MR 2023. The ER G1 uses less than 1/3 of the JBD’s display’s 640×480 pixels.

ER’s G1 Uses Less Than 1/3rd of the JBD’s Display’s Pixels

In looking at both Even Realities “simulated” and actual through-the-optics images, I was struck by the lack of content and wide and short aspect ratio. ER G1 displays about a 3.6 to 1 versus the display’s 4:3 aspect ratio or 1.333 to 1. So I decided to investigate. The best through-the-optics image I could find online was in a Jon Rettinger YouTube review (right).

Jon Rettinger’s picture was taken handheld with a smartphone and was not aligned and focused well. Still, I could make out some pixels and other details. I use a very high-resolution picture through a different waveguide of the JBD 640×480 green MicroLED to “calibrate” the pixel sizes in Regittinger’s ER G1 through the optics picture.

In the comparison below, I have removed (most) of the geometric distortion in the picture, enhanced the image slightly to see the pixels better, and scaled the image to match the scale of the higher-resolution image of the whole JBD display. The net result is that it looks like the ERG1 is only displaying about 550 by 150 pixels, or ~86% of the horizontal pixels and ~31% of the possible vertical pixels.

Even Realities demos use larger fonts than many others (ex., see the Oppo image shown earlier). They seem to be trying to keep the information presented to the user in small, easily digested amounts.

Application Examples From Even Realities

Even Realities’ website and videos show application examples that appear to be “simulated” (few mark their images as simulated anymore). It looks like the virtual display image (which is a little “too good”) is being composited on a computer on top of a photograph or video, not through optics. Still, the level of content and resolution is consistent with the capabilities demonstrated in the through-the-optics videos and pictures I have found.

Some of the applications are shown on the right. The teleprompter mode uses “AI” to scroll the text” and fade words that have been spoken.

They have classic “notifications.” displays. Like every other “AI” glasses, both audio-only and with displays, the ER G1 plans to support language translation. Language translation should work much better with a display than with audio translation alone (which was discussed in our panel at AWE 2024).

The other well-known potential application for AR glasses has been navigation. In this area, the limitations due to the lack of resolution and color seem obvious compared to smartphones or smartwatches. It might work well with a cell phone map to limit your need to refer to the smartphone. Caution: Driving with AR glasses, at least turned on, today would probably be dangerous (and perhaps illegal in many states and countries). So this application is for people who are on foot or as passengers.

Below are some frame captures from TT Technologies’ YouTube video taken through the optics. As stated above, the content level is consistent with the simulated images by Even Realities.

A key issue with all these applications is whether they will provide enough functionality for regular/everyday use in a world with smartwatches and smartphones. For example, I typically fly out of the country once or twice a year, and the translation might be useful. With a smartphone, I can use it for translation, including taking pictures of signage and translating them (something the G1 can do since it does not have cameras) on my smartphone. It’s not worth it to me to pay ~$750 for another product to do AI translation; I can look down at my smartphone for the few occasions I need it. It might be a different case for someone immersed for a long time in a foreign country or an executive who wants to know what someone is saying in a foreign language in a business meeting.

No Cameras, Gesture/SLAM, Videos, and Pictures

The ER G1 is a minimalist approach to AI/AR glasses. Most obviously missing are cameras found in audio-only AI Glasses, such as the Meta Ray Ban Wayfarer. The only input to the “AI” is via audio. Lacking video eliminates a whole set of proposed “AI” based applications. However, adding cameras to the G1 would mean more space, weight, and power due to the cameras and the need for higher data rates back to the smartphone. Adding cameras would likely push the weight above 40 grams (Even Realities has not given a precise number but has said the G1 is less than 40 grams).

The “Glassholes” argument about not having a camera on AR glasses is overblown. Cameras are everywhere these days and I think it was just a lazy excuse. There were many other problems with Google Glass, including the looks, display performance, and functionality, not to mention the vastly overblown marketing that grossly overstated their capability.

In the “normal glasses,” form factor and weight are out of the question, as are supporting gesture recognition and SLAM. The G1 relies on a capacitive touch sensor and voice recognition.

With a very low-resolution green-only image and a low data rate, the G1 is unsuitable for watching movies or looking at pictures. I’m not sure the G1 would even support pictures and movies. It’s a very bare-bones data-snacking AR device.

Conclusions

I must credit Even Realities for adhering to their minimalist principles and developing consistent features. It is an example of the trade-offs that must be made to keep AR glasses under 40 grams.

I also appreciate why (I think) they chopped the waveguide and, therefore, the display height to prevent the diffraction grating from interfering with the direct forward view. Each alternative display location to the G1 has a different set of drawbacks. I find the thought experiment of where to put the display in the FOV and the resultant issues illustrative of how tight a balance must be with AR glasses.

Integrating (gluing) prescription lenses into glasses has advantages for an all-day wearable product. But it is not without its problems, including the issue of what happens when a user’s prescription changes and the ability for multiple people to try the same AR glasses.

When you buy the product plus prescriptions, the price is about $750 (or maybe more for more complex prescriptions). What you are getting is a low-resolution green-only display that, other than being always available in front of you, can’t compete with a smartwatch in terms of display capability.

For a consumer product, full color is more important than it would be for an “enterprise” product. As a leader of IBM’s computer monitor division said in the early 1980s during a debate over higher-resolution monochrome (up to ~2K by ~1K) versus lower-resolution color monitors (~640×480 at that time), “Color is the least necessary and most desired feature in a monitor.” Monochrome displays are often still used in military and industrial applications. And we saw this again as smartwatches moved from monochrome to color. Consumers still much prefer color even if it is not “necessary.” Even with low-resolution, more information displays, color has functional advantages including simple things such as red=bad/warning and green=good.

Even Realities says that their presales are greatly exceeding expectations. We will have to wait and see if this keeps up as the product goes into production and whether users find it useful. Everyone in this field expects Meta and many others to introduce “AI” AR (with display) Glasses that support audio and camera input and audio and display output. So we are about to see the “experiment run” with various design features and trade-offs. As stated previously, I understand that cameras would drive up the weight and power, but of all the trade-offs, Even Realities made with the G1, leaving out camera(s) is the issue I would most second guess.

Introduction

Based on information gathered at SID Display Week and AWE, I have many articles to write based on the thousands of pictures I took and things I learned. I have been organizing and editing the pictures.

As its name implies, Display Week is primarily about display devices. My major takeaway from that conference is that many companies work on full-color MicroLEDs with different approaches, including quantum dot color conversion, stack layers, and single emitter with color shifting based on current or voltage.

AWE moved venues from the Santa Clara Convention Center in Silicon Valley to the larger Long Beach Convention Center south of LA. More than just a venue shift, I sensed a shift in direction. Historically, at AWE, I have seen many optical see-through AR/MR headsets, but there seem to be fewer optical headsets this year. Instead, I saw many companies with software running on VR/Passthrough AR headsets, primarily on the Meta Quest 3 (MQ3)and Apple Vision Pro (AVP).

This article was partly inspired by Hypervision’s white paper discussing whether micro-OLEDs or small LCDs were the best path to 60 pixels per degree (PPD) with a wide FOV combined with the pictures I captured through Hypervision’s HO140 (140° diagonal FOV per eye) optics at AWE 2024. I have taken thousands of pictures through various headsets, and the Hypervision picture stood out in terms of FOV and sharpness. I have followed Hypervision since 2021 (see Appendix: More on Hypervision).

I took my first pictures at AWE through the Sony XR (SXR) Headset optics. At least subjectively, in a short demo, the SXR’s image quality (sharpness and contrast) seemed higher than that of the AVP, but the FOV was smaller. I had on hand (thousands) of pictures I had taken through the Big Screen Beyond (BSB), AVP, Meta Quest Pro (MQP), and Meta Quest 3 (MQ3) optics with the same camera and lens, plus a few of the Hypervision HO140 prototype. So, I decided to make some comparisons between various headsets.

I also want to mention LightPolymers’ new Quarter Waveplate (QWP) and Polarization technologies, which I first learned about from a poster in the Hypervision AWE booth. In April 2024, the two companies announced a joint development grant. They offer an alternative to the plastic film QWP and Polarizers, where 3M dominates today.

Hypervision’s HO140 Display

Based on my history of seeing Hypervision’s 240° prototypes for the last three years, I had, until AWE 2024, largely overlooked their single display 140° models. I had my Canon R5 (45Mp with 405mp ” 3×3 sensor pixel shift mode”) and tripod with me at AWE this year, so I took a few high-resolution pictures through the optics of the HO140. Below are pictures of the 240° (left) and 140° (right) prototypes in the Hypervsion Booth. Hypervision is an optics company and not a headset maker and the demos are meant to show off their optics.

When I got home and looked at the pictures through the HO140, I was impressed by the overall image quality of the HO140, after having taken thousands of pictures through the Apple Vision Pro (with Micro-OLED displays) and Meta’s Quest Pro, Quest 3 (both with mini-LCD displays), the Big Screen Beyond. It usually takes me considerable time and effort, as well as multiple reshoots, to find the “sweet spot” for the other devices, but I got good pictures through the HO140 with minimal effort and only a few pictures, which suggests a very large sweet spot in Hypervision’s optical design. The HO140 is a prototype of unknown cost that I am comparing to production products. I only have this one image to go by and not a test pattern.

The picture below is from my Canon R5, with a 16mm lens netting a FOV of 97.6° horizontal by 73.7° vertical. It was shot at 405mp and then reduced to 45mp to avoid moiré effects due to the “beat frequencies” between the camera sensor and the display devices with their color subpixels. All VR optics pincushion, which causes the pixel sizes to vary across the display and increases the chance of getting moiré in some regions.

The level of sharpness throughout the HO140’s image relative to other VR headsets suggests that it could support a higher-resolution LCD panel with a smaller pixel size if it existed. Some significant chroma aberrations are visible in the outer parts of the image, but these could be largely corrected in software.

Compared to other VR-type headsets I have photographed, I was impressed by how far out into the periphery of the FOV the image maintains sharpness while supporting a significantly larger FOV than any other device I have photographed. What I can’t tell without being able to run other content, such as test patterns, is the contrast of the display and optics combination.

I suggest also reading Hypervision’s other white papers on their Technology & Research page. Also, if you want an excellent explanation of pancake optics, I recommend Arthur Rabner’s, CTO of Hypervision, one-hour and 25-minute presentation on YouTube.

Sony XR (SXR)

Mechanical Ergonomics

AWE was my first time trying the new Sony XR (SXR) headset. In my CES 2024 coverage, I wrote about the ergonomic features I liked in Sony XR (and others compared to Apple Vision Pro). In particular, I liked the headband approach with the flip-up display, and my brief try with the Sony headset at AWE seemed to confirm the benefits of this design choice (which is very similar to the Lynx R1 headset), at least from the ergonomics perspective relative to the Apple Vision Pro.

Still, the SXR is still pretty big and bulky, much more so than the AVP or Lynx. Having only had a short demo, I can’t say how comfortable it will be in extended use. As was the case for the HO140, I couldn’t control the content.

“Enterprise” Product

Sony has been saying that this headset primarily aims at “enterprise” (= expensive high-end) applications, and they partner with Siemens. It is much more practical than the Apple Vision Pro (AVP). The support on the head is better; it supports users wearing their glasses, and the display/visor flips up so you can see the real world directly. There is air circulation to the face and eyes. The headset also supports adjustment of the distance from the headset to the eyes. The headset allows peripheral vision but does have a light shield for full VR operation. The headset is also supposed to support video passthrough, but that capability was not demonstrated. As noted in my CES article, the SXR headset put the pass-through cameras in a much better position than the AVP.

Display Devices and Image Quality

Both the AVP and SXR use ~4K micro-OLED display devices. While Sony does the OLED Assembly (applying the OLED and packaging) for its headset and the AVP’s display devices, the AVP reportedly uses a custom silicon backplane designed by Apple. The SXR’s display has ~20% smaller 6.3-micron pixels than the AVP’s 7.5-micron. The device size is also smaller. The size factors of the SXR favor higher angular resolution and a smaller FOV, as is seen with the SXR.

The picture below was taken (handheld) with my 45MP Canon R5 camera with a 16mm lens like the HO140, but because I couldn’t use a tripod, I couldn’t get a 405MP picture with the camera’s sensor shifting. I was impressed that I got relatively good images handheld, which suggests the optics have a much larger sweet spot than the AVP, for example. To get good images with the AVP requires my camera lens to be precisely aligned into the relatively small sweep spot of the AVP’s optics (using a 6-degree-of-freedom camera rig on a tripod). I believe the Apple Vision Pro’s small sweet spot and the need for eye-tracking-based lens correction, and not just for foveated rendering, are part of why the AVP has to be uncomfortably clamped against the user’s face.

Given that I was hand-holding both the headset and camera, I was rather surprised that the pictures came out so well (click on the image to see it in higher, 45mp resolution).

At least in my brief demo, the SXR’s optics image quality seems better than the AVP’s. The images seem sharper with lesser chroma (color) aberrations. The AVP seems heavily dependent on eye tracking to correct optics problems with the optics, but it does not always succeed.

Much more Eye Refief (enabling eye glasses) but lower FOV

I was surprised by how much eye relief the SXR optics afforded compared to the AVP and BSB, which also use Micro-OLED microdisplays. Typically, the requirement for high magnification of the micro-OLED pixels compared to LCD pixels inherently makes eye relief more difficult. The SXR magnifies less, resulting in a smaller FOV, but also makes it easier optically for them to support more eye relief. But note, taking advantage of the greater eye relief will further reduce the FOV. The SXR headset has a smaller FOV than any other VR-type headset I have tried recently.

Novel Sony controllers were not a hit

While I will credit Sony for trying something new with the controllers, I didn’t like finger trackpad and ring color are great solutions. I talked with several people who tried them, and no one seemed to like either controller. It is hard to judge control devices in a short demo; you must work with them for a while. Still, they didn’t make a good first impression.

VR Headset “Shootout” between AVP, MQP, Big Screen Beyond, Hypervision, and Sony XR

I have been shooting VR headsets with the Canon R5 with a 16mm lens for some time and built up a large library of pictures. For the AVP, Big Screen Beyond (BSB), and Meta Quest Pro (MQP), I had both the the headset and the camera locked down on tripods so I could center the lens in the sweet spot of the optics. For the Hypervision, while the camera and headset were on tripods, my camera was only on a travel tripod without my 6-degree-of-freedom rig and the time to precisely locate the headset’s optical sweet spot. The SXR picture was taken with my hand holding the headset and the camera.

Below are through-the-optics pictures of the AVP, BSB, MQP, Hypervision HO140, and SXR headsets, all taken with the same camera and lens combination and scaled identically. This is not a perfect comparison as the camera lens does not work identically to the eye (which also rotates), but it is reasonably close. The physically shorter and simpler 16mm prime (non-zoom) lens lets it get inside the eye box of the various headsets for the FOV it can capture.

FOV Comparison (AVP, SXR, BSB, HO140, MQ3/MQP)

While companies will talk about the number of horizontal and vertical pixels of the display device, the periphery of the display’s pixels are cut off by the optics, which tend to be circular. All the VR headset optics have a pincushion distortion, which results in higher resolution in the sweet spot (optical center), which is always toward the nose side and usually above the center for VR headsets.

In the figure below, I have overlaid the FOV of the left eye for the headsets on top of the picture HO140 image. I had to extrapolate somewhat on the image circles on the top and bottom as the headset FOVs exceeded the extent of the camera’s FOV. The HO140 supports up to a 2.9″ diagonal LCD (that does not exist yet), but they currently use a 2.56″ 2160×2160 Octagonal BOE LCD and are so far beyond the FOV of my camera lens that I used their information.

As can be seen, the LCD-based headsets of Hypervision and Meta typically have larger FOV than the micro-OLED-based headsets of AVP, Meta, and Sony. However, as will be discussed, the micro-OLED-based headsets have smaller pixels (angularly and on the physical display device).

Center Pixels (Angular Size in PPD)

Due to handholding the SXR and having pixels smaller than the AVP, I couldn’t get a super-high-resolution (405 mp) image from the center of the FOV and didn’t have the time to use a longer focal length lens to show the pixel boundaries. The SXR has roughly the same number of pixels as the AVP but a smaller FOV, so its pixels are angularly smaller than the AVP’s. I would expect the SXR to be near 60 pixels per degree (PPD) in the center of the FOV. The BSB has about the same FOV as the AVP but has a ~2.5K micro-OLED compared to the AVP’s ~4K; thus, the BSB pixels in the center are about 1.5x bigger (linearly). The Hypervision’s display has a slightly smaller center pixel pitch than the MQP (and MQ3) but with a massively bigger FOV.

The MQP (and the very similar MQ3) rotate the display device. To make it easier to compare the pixel pitches, I included a rotated inset of the MQP pixels to match the alignment of the other devices. Note that the pictures below are all “through the optics” and thus include the headset’s optical magnification. I have given the angular resolution in PPD for each headset. I have indicated the angular resolution (in pixels-per-degree, PPD) for each of the headset’s center pixels. For the center pixels pictures below, I used a 28mm lens to get more magnification to see sub-pixel detail for the AVP, BSB, and MQP. I only took 16mm lens pictures of the HO140 and, therefore, rescaled the image based on the different focal lengths of the lens.

The Micro-OLED base headsets require significantly more optical magnification than the LCD models. For example, the AVP has 3.2x (linearly) smaller display device pixels than the MQP, but after optics, the pixels are ~1.82x smaller. As a specific example, the AVP magnifies the display by ~1.76 more than the MQP.

Outer Pixels

I capture pixels from a similar (very approximately) distance from the optical center of the lens. The AVP’s “foveated rendering” makes it look worse than it is, but you can still see the pixel grid with the others. Of the micro-OLED headsets, the BSB and SXR seem to do the best regarding sharpness in the periphery. The Hypervision HO140 pixels seem much less distorted and blurry than any of the headsets, including the MQP and MP3, which have much smaller FOVs.

Micro-OLED vs. Mini-LCD Challenges

Micro-OLEDs are made by applying OLEDs on top of a CMOS substrate. CMOS transistors provide a high current per unit area, and all the transistors and circuitry are underneath the OLED pixels, so it doesn’t block light. These factors enable relatively small pixels of 6.3 to 10 microns. However, CMOS substrates are much more expensive per unit area, and modern semiconductor FABs limit of CMOS devices is about 1.4-inch diagonal (ignoring expensive and low-yielding “reticle stitched” devices).

A basic issue with OLEDs is that the display device must provide the power/current to drive each OLED. In the case of LCDs, only a small amount of capacitance has to be driven to change the pixel, after which there is virtually no current. The table on the right (which I discussed in 2017) shows the transistor mobility and the process requirements for the transistors for various display backplanes. The current need for an emitting display device like OLEDs and LEDs requires crystalline silicon (e.g., CMOS) or much larger thin-film transistors on glass. There are also issues of the size and resistivity of the wires used to provide the current and heat issues.

The OLED’s requirement for significant current/power limits how small the pixels can get on a given substrate/technology. Thin-film transistors have to be physically big to supply the current. For example, the Apple Watch Ultra Thin Film transistor OLED display has 326 PPI (~78 microns), which is more than 10x larger linearly (100x the area) than the Apple Vision Pro’s pixel, even though both are “OLEDs.”

Another issue caused by trying to support large FOVs with small devices is that the higher magnification reduces eye relief. Most of the “magnification” comes from moving the device closer to the eye. Thus, LCD headsets tend to have more eye relief. Sony’s XR headset is an exception because it has enough eye relief for glasses but does so with a smaller FOV than the other headsets.

Small LCDs used in VR displays have different challenges. They are made on glass substrates, and the transistors and circuitry must be larger. Because they are transmissive, this circuitry in the periphery of each pixel blocks light and causes more of a screen door effect. The cost per unit area is much lower than that of CMOS, and LCD devices can be much larger. Thus, less aggressive optical magnification is required for the same FOV with LCDs.

LCDs face a major challenge in making the pixels smaller to support higher resolution. As the pixels get smaller, the size of the circuitry relative to the pixel size becomes bigger, blocking more light and causing a worse screen door effect. To make the pixels smaller, they must develop higher-performance thin-film transistors and lower resistance interconnection to keep blocking too much light. This subject is discussed in an Innolux Research Paper published by SPIE in October 2023 (free to download). Innolux discusses how to go from today’s typical “small” LCD pixel of 1200 ppi (=~21 microns) to their research device with 2117 ppi (=~12 microns) to achieve a 3840 x 3840 (4K by 4k) display in a 2.56″ diagonal device. Hypervision’s HO140 white paper discusses Innolux’s 2022 research prototype with the same pixel size but with 3240×3240 pixels and a 2.27-inch panel, as well as the current prototype. The current HO140 uses a BOE 2.56″ 2160×2160 panel with 21-micron pixels, as the Innolux panel is not commercially available.

Some micro-OLED and small LCD displays for VR

YouTuber Brad Lynch of SadlyItsBradley, in an X post, listed the PPI of some common VR headset display devices. I have added more entries and the pixel pitch in microns. Many VR panels are not rectangular and may have cut corners on the bottom (and top). The size of the panels given in inches is for the longest diagonal. As you can see, Innolux’s prototypes have significantly smaller pixels, but almost 2x linearly, than the VR LCDs in volume production today:

• Vive: 3.6″, 1080p, ~360 PPI (70 microns)
• Rift S*: 5.5″, 1280P, ~530 PPI (48 microns)
• Valve Index: 3.5″, 1440p, ~600 PPI (42 microns)
• Quest 2*: 5.5″, 1900p, ~750 PPI (34 microns)
• Quest 3: ~2.55″ 2064 × 2208, 1050 PPI (24 microns) – Pancake Optics
• Quest Pro: 2.5″, 1832×1920, ~1050 PPI (24 microns) – Might be BOE 2.48″ miniLED LCD
• Varjo Aero: 3.2″, 2880p, ~1200 PPI (21 microns)
• Pico 4: 2.5″, 2160p, 1192 PPI (21 microns)
• BOE 2.56″ LCD, 2160×2160, 1192 PPI (21 microns) – Used in Hypervision HO140 at AWE 2024
• Innolux 2023 Prototype 2.56″, 3840×3840, 2117 ppi (12 microns) -Research prototype
• Apple Vision Pro 1.4″ Micro-OLED, 3,660×3,200, 3386 PPI (7.5 microns)
• SeeYa 1.03″ Micro-OLED, 2560×2560, 3528 PPI (7.2 microns) – Used in Big Screen Beyond
• Sony ~1.3″ Micro-OLED, 3552 x 3840, 4032 PPI (6.3 microns) – Sony XR
• BOE 1.35″ Micro-OLED 3552×3840, 4032 PPI (6.3 microns) – Demoed at Display Week 2024

In 2017, I wrote Near Eye Displays (NEDs): Gaps In Pixel Sizes (table from that article on the right) talks about what I call the pixel size gap between microdisplays (on Silicon) and small LCDs (on glass). While the pixel sizes have gotten smaller for both micro-OLED and LCDs for VR in the last ~7 years, there remains a sizable gap.

Contrast – Factoring the Display and Pancake Optics

Micro-OLEDs at the display level certainly have a better inherent black level and can turn pixels completely off. LCDs work by blocking light using cross-polarization, which results in imperfect blacks. Thus, with micro-OLEDs, a large area of black will look black, whereas with LCDs, it will be dark gray.

However, we are not looking at the displays directly but through optics, specifically pancake optics, which dominate new VR designs today. Pancake optics, which use polarized light and QWP to recirculate the image twice through parts of the optics, are prone to internal reflections that cause “ghosts” (somewhat out-of-focus reflections) and contrast loss.

Using smaller micro-OLEDs requires more “aggressive” optical designs that support higher magnification to support a wide FOV. These more aggressive optical designs can be more prone to being more expensive, less sharp, and loss of polarization. Any loss of polarization in pancake optics will cause a loss of contrast and ghosting. There seems to be a tendency with pancake optics for the stray light to bounce around and end up in the periphery of the image, causing a glow if the periphery of the image is supposed to be black.

For example, the AVP is known to have an outer “glow” when watching movie content on a black background. Most VR headsets default to a “movie or home theater” rather than a background. While it may be for aesthetics, the engineer in me thinks it might help hide the glow. People online suggest turning on some background with the AVP for people bothered by the glow on a black background.

The complaints of outer glow when watching movies seem more prevalent when using headsets micro-OLEDs, but this is hardly scientific. It could be just that the micro-OLEDs have a better black level and make the glow more noticeable, but it might also be caused by their more aggressive optical magnification (something that might be or has been (?) studied). My key point is that it is not as simple as considering the display’s inherent contrast, you have to consider the whole optical system.

LightPolymers’ Alternative to Plastic Films for QWP & Polarizers

LightPolymers has a Lyotropic (water-based) Liquid Crystal (LC) material that can make optical surfaces like QWP and polarizers. Silicon Optix, which the blog broke the news of Meta buying them in December 2021 (Exclusive: Imagine Optix Bought By Meta), was also developing LC-based polarized light control films.

Like Silicon Optix, Light Polymers has been coating plastic films with LCs, but LightPolymers is developing the ability to directly apply their films to flat and curved lenses, which is a potential game changer. In April 2024, LightPolymers and Hypervision announced the joint development of this lens-coating technology and had a poster in their Hypervision’s booth showing it (right)

3M Dominates Polarized Light Plastic Films for Pancake Optics

3M is today the dominant player in polarized light-control plastic films and is even more dominant in these films for pancake optics. At 3M’s SID Display Week booth in June 2024, they showed the ByteDance PICO4, MQP, and MQ3 pancake optics using 3M polarization films. Their films are also used in the Fresnel lens-based Quest 2. It is an open secret (but 3M would not confirm or deny) that the Apple Vision Pro also uses 3M polarization films.

According to 3M:

“3M did not invent the optical architecture of pancake lenses. However, 3M was the first company to successfully demonstrate the viability of pancake lenses in VR headsets by combining it with its patented reflective polarizer technology.“

That same article supports Kopin’s (now spun out to Lightning Silicon) claims to have been the first to develop pancake optics. Kopin has been demonstrating pancake optics combined with their Micro-OLEDs for years, which are used in Panasonic-ShiftAll headsets.

3M’s 2017 SPIE Paper Folded Optics with Birefringent Reflective Polarizers discusses the use of their films (and also mentions Kopin developments) in cemented (e.g., AVP) and air gap (e.g., MQP and MP3) pancake optics. The paper also discusses how their polarization films can be made (with heat softening) to conform to curved optics such as the AVP.

LightPolymers’ Potential Advantage over Plastic Films

The most obvious drawbacks of plastic films are that they are relatively thick (on the order of 70+ microns per film, and there are typically multiple films per lens) and are usually attached using adhesive coatings. The thickness, particularly when trying to conform to a curved surface, can cause issues with polarized light. The adhesives introduce some scatter, resulting in some loss of polarization.

By applying their LCs directly to the lens, LightPolymer claims they could reduce the thickness of the polarization control (QWP and Polarizers) by as much as 10x and would eliminate the use of adhesives.

In the photos below (taken with a 5x macro lens), I used a knife to slightly separate the edges of the films from the Meta Quest 3’s eye-side and display-side lenses to show them. On the eye-side lens, there are three films, which are thought to be a QWP, absorptive polarizer, and reflective polarizer. On the display-side lens, there are two films, one of which is a QWP, and the other may be just a protective film. In the eye-side lens photo, you can see where the adhesive has bubbled up after separation. The diagram on the right shows the films and paths for light with the MQ3/MQP pancake optics.

Because LighPolymers’ LC coating is applied to each lens, it could also be applied/patterned to improve or compensate for other issues in the optics.

Current State of LightPolymer’s Technology

LightPolymers is already applying its LC to plastic films and flat glass. Their joint agreement with Hypervision involves developing manufacturable methods for directly applying the LC coatings to curved lens surfaces. This technology will take time to develop. LightPolymer business of making the LC materials and then works with partners such as Hypervision to apply the LC to their lenses. They say the equipment necessary to apply the LCs is readily available and low-cost (for manufacturing equipment).

Conclusion

Hypervision has demonstrated the ability to design very wide FOV pancake optics with a large optical sweet spot and maintains a larger area of sharpness than any other design I have seen.

Based on my experience in both Semiconductors and Optics, I think Hypervision makes a good case in their white paper 60PPD: by fast LCD but not by micro OLED, getting to a wide FOV while approaching “retinal” 60PPD is more likely to happen using LCD technology than micro-OLEDs.

Fundamentally, micro-OLEDs are unlikely to get much bigger than 1.4″ diagonally, at least commercially, for many years, if not more than a decade. While they could make the pixels smaller, today’s pancake optics struggle to resolve ~7.5-micron pixels, no less small ones.

On the other hand, several companies, including Innoulux and BOE, have shown research prototypes of 12-micron LCD pixels, or half the (linear) size of today’s LCDs used in VR headsets in high volume. If BOE or Innolux went into production with these displays, it would enable Hypervision’s HO140 to reach about 48 PPD in the center with a roughly 140-degree FOV, and only small incremental changes would get them to 60 PPD with the same FOV.

Appendix: More on Hypervision

I first encountered Hypervision at AWE 2021 with their blended Fresnel lens 240-degree design, but as this blog primarily covered optical AR, it slipped under my radar. Since then, I have been covering Optical and Pass-Through mixed reality, particularly pass-through MR using Pancake Optics. By AR/VR/MR 2023, Hypervsion demonstrated a single lens (per eye) 140-degree and a blended dual lens and display 240-degree FOV (diagonal) Pancake Optics designs.

These were vastly better than their older Fresnel designs and demonstrated Hypervision’s optical design capability. In May 2023, passthrough MR startup Lynx and Hypervision announced they were collaborating. For some more background on my encounters with Hypervision, see Hypervision Background.

Hypervision has been using its knowledge of pancake optics to analyze the Apple Vision Pro’s optical design, which I have reported on in Hypervision: Micro-OLED vs. LCD – And Why the Apple Vision Pro is “Blurry,” Apple Vision Pro Discussion Video by Karl Guttag and Jason McDowall, Apple Vision Pro – Influencing the Influencers & “Information Density,” and Apple Vision Pro (Part 4)—Hypervision Pancake Optics Analysis.

Introduction

At AWE 2024, I was on a panel discussion titled “The Current State and Future Direction of AR Glasses.” Jeri Ellsworth, CEO of Tilt Five, Ed Tang, CEO of Avegant, Adi Robertson, Senior Reporter at The Verge, and I were on the panel, with Jason McDowell, The AR Show, moderating. Jason McDowell did an excellent job of moderation and keeping the discussion moving. Still, with only 55 minutes, including questions from the audience, we could only cover a fraction of the topics we had considered discussing. I’m hoping to reconvene this panel sometime. I also want to thank Dean Johnson, Associate Professor at Western Michigan University, who originated the idea and helped me organize this panel. AWE’s video of our panel is available on YouTube.

First, I will outline what was discussed in the panel. Then, I want to follow up on small FOV optical AR glasses and some back-and-forth discussions with AWE Legend Thad Starner.

Outline of the Panel Discussion

The panel covered many topics, and below, I have provided a link to each part of our discussion and added additional information and details for some of the topics.

• 0:00 Introductions
• 2:19 Apple Vision Pro (AVP) and why it has stalled. It has been widely reported that AVP sales have stalled. Just before the conference, The Information reported that Apple had suspended the Vision Pro 2 development and is now focused on a lower-cost version. I want to point out that a 1984 128K Mac 1 adjusted for inflation would cost over $7,000 adjusted for inflation, and the original 1977 Apple 2 4K computer (without a monitor or floppy drive) would cost about $6,700 in today’s dollars. I contend that utility and not price is the key problem with the AVP sales volume and that Apple is thus drawing the wrong conclusion.
• 7:20 Optical versus Passthrough AR. The panel discusses why their requirements are so different.
• 11:30 Mentioned Thad Starner and the desire for smaller FOV optical AR headsets. It turns out that Thad Starner attended our panel, but as I later found out, he arrived late and missed my mentioning him. Thad, later questioned the panel. In 2019, I wrote the article FOV Obsession, which discussed Thad’s SPIE AR/VR/MR presentation about smaller FOV. Thad is a Georgia Institute of Technology professor and a part-time Staff Researcher at Google (including on Google Glass). He has continuously worn AR devices since his research work at MIT’s media lab in the 1990s.
• 13:50 Does “tethering make sense” with cables or wirelessly?
• 20:40 Does an AR device have to work outside (in daylight)?
• 26:49 The need to add displays to today’s Audio-AI glasses (ex. Meta Ray-Ban Wayfarer).
• 31:45 Making AR glasses less creepy?
• 35:10 Does it have to be a glasses form factor?
• 35:55 Monocular versus Biocular
• 37:25 What did Apple Vision Pro get right (and wrong) regarding user interaction?
• 40:00 I make the point that eye tracking and gesture recognition on the “Apple Vision Pro is magical until it is not,” paraphrasing Adi Robertson, and I then added, “and then it is damn frustrating.” I also discuss that “it’s not truly hands-free if you have to make gestures with your hands.”
• 41:48 Waiting for the Superman [savior] company. And do big companies help or crush innovation?
• 44:20 Vertical integration (Apple’s big advantage)
• 46:13 Audience Question: When will AR glasses replace a smartphone (enterprise and consumer)
• 49:05 What is the first use case to break 1 million users in Consumer AR?
• 49:45 Thad Starner – “Bold Prediction” that the first large application will be with small FOV (~20 degrees), monocular, and not centered in the user’s vision (off to the ear side by ~8 to 20 degrees), and monochrome would be OK. A smartphone is only about 9 by 15 degrees FOV [or ~20 degrees diagonally when a phone is held at a typical distance].
• 52:10 Audience Question: Why aren’t more companies going after OSHA (safety) certification?

Small FOV Optical AR Discussion with Thad Starner

As stated in the outline above, Thad Starner arrived late and missed my discussion of smaller FOVs that mentioned Thad, as I learned after the panel. Thad, who has been continuously wearing AR glasses and researching them since the mid-1990s, brings an interesting perspective. Since I first saw and met him in 2019, he has strongly advocated for AR headsets having a smaller FOV.

Thad also states that the AR headset should have a monocular (single-eye) display and be 8—to 20 degrees on the ear side of the user’s straight-ahead vision. He also suggests that monochrome is fine for most purposes. Thad stated that his team will soon publish papers backing up these contentions.

In the sections below, I went from the YouTube transcript and did some light editing to make what was said more readable.

My discussion from earlier in the panel:

11:30 Karl Guttag – I think a lot of the AR or Optical see-through gets confabulated with what was going on in VR because VR was cheap and easy to make a wide field of view by sticking a cell phone with some cheap Optics in front of your face. You get a wide field of view, and people went crazy about that. I made this point years ago on my blog [2019 article FOV Obsession] was the problem. Thad Starner makes this point: he’s one of our Legends at AWE, and I took that to heart many years ago at SPIE AR/VR/MR 2019.

The problem is that as soon as you say beyond about 30-degree field of view, even projecting forward [with technology advancements], as you go beyond 30-degree field of view, you’re in a helmet, something looking like Magic Leap. And Magic Leap ended up in Nowheresville. [Magic Leap] ended up with 25 to 30% see-through, so it’s not really that good see-through, and yet it’s not got the image quality that you would get of an old display shot right in your eyes. You might you could get a better image on an Xreal or something like that.

People are confabulating too many different specs, so they want a wide field of view. The problem is as soon as you say 50 degrees and then you say, yeah, and I need like spatial recognition, I want to do SLAM, and I want to do this, and I want to do that. You’ve now spiraled into the helmet. I mean, you know, Meta was talking the other day about the other panels and said they’re looking at about 50 grams [for the Meta Ray Bans], and my glasses are 23 grams. You’re out of that as soon as you say 50-degree field of view, you’re over 100 grams and and and and and heading to the Moon as you add more and more cameras and all this other stuff, so I think that’s one of our bigger problems whereas AR really Optical AR.

The experiment we’re going to see played out because many companies are working on adding displays to to so called AI audio glasses. We’re going to see if that works because companies are getting ready to make glasses that have 20—to 30-degree field of view glasses tied into AI and audio stuff.

Thad Starner’s comments and the follow-up discussion during the Q&A at the end of the panel:

49:46 Hi, my name is Thad Starner. I’m Professor Georgia Tech. I’m going to make a bold prediction here that the future, at least the first system to sell over a million units, will be a small field of view monocular, non-line-of-sight display, monochrome is okay now; the reason I say that is number one I’ve done different user studies in my lab that we’ll be publishing soon on this subject but the other thing is that you know our phones which is the most popular interface out there are only 9 degrees by 16 degrees field of view. Putting something outside of the line of sight means that it doesn’t interrupt you while you’re crossing the street or driving or flying a plane, right? We know these numbers, so between 8° and 20 degrees towards the ear and plus or minus 8 degrees, I’m looking at Karl [Guttag] here so he can digest all these things.

Karl – I wrote a whole article about it [FOV Obsession]

Thad – And not having a pixel in line of sight, so now feel free to pick me apart and disagree with me.

Jeri-  I want to know a price point.

Thad, I think the first market will be captioning for the heart of hearing, not for the deaf. Also, possible transcription, not translation; at that price point, you’re talking about making reading glasses for people instead of hearing aids. There’s a lot of pushback against hearing, but reading glasses people tend to do, so I’d say you’re probably in the $200 to $300 range.

Ed – I think your prediction is spot on, minus the color green. The only thing I think is that it’s not going to fly.

Thad – I said monochrome is okay.

Ed – I think the monocular field of view is going to be an entry-level product, and you see, I think you will see products that will fit that category with roughly that field of view with roughly that offset angle [not in the center of view] is what you’re going to see in the beginning. Yeah I agree with that but I don’t I think that’s the first step I think you will see a lot of products after that that’s going to do a lot more than monocular monochrome offset displays, start going to larger field of view binocular I think that will happen pretty quickly.

Adi – It does feel like somebody tries to do that every 18 months, though, like Intel tried to make a pair of glasses that did that. It’s a little bit what North did. I guess it’s just a matter of throwing the idea at the wall because I think it’s a good one until it takes.

I was a little taken aback to have Thad call me out as if I had disagreed with him when I had made the point about the advantages of a smaller FOV earlier. Only after the presentation did I find out that he had arrived late. I’m not sure what comment I made that made Thad think I was advocating for a larger FOV in AR glasses.

I want to add that there can be big differences between what consumers and experts will accept in a product. I’m reminded of a story I read in the early 1980s when there was a big debate between very high-resolution monochrome versus lower-resolution color (back then, you could only have one or the other with CRTs) that the head of IBM’s monitor division said, “Color is the least necessary and most desired feature in a monitor.” All the research suggested that resolution was more important for the tasks people did on a computer at the time, but people still insisted on color monitors. Another example is the 1985 New Coke fiasco, in which Coke’s taste studies proved that people liked New Coke better, but it still failed as a product.

In my experience, a big factor is whether the person is being trained to use the device for enterprise or military use versus whether the user is buying it for their own enjoyment. The military has used monochrome displays on devices, including night vision and heads-up displays for decades. I like to point out that the requirement can change if “If the user paid to use versus is paying to use.” Enterprises and the military care about whether the product gets the job done and pay someone to use the device. The consumer has different criteria. I will also agree that there are cases where the user is motivated to be trained, such as Thad’s hard-of-hearing example.

Conclusion on Small FOV Optical AR

First, I agree with Thad’s comments about the smaller FOV and have stated such before. There are also cases outside of enterprise and industrial use where the user is motivated to be trained, such as Thad’s hard-of-hearing example. But while I can’t disagree with Thad or his studies that show having a monocular monochrome image located outside the line of sight is technically better, I think consumers will have a tougher time accepting a monocular monochrome display. What you can train someone to use differs from what they would buy for themselves.

Thad makes a good point that having a biocular display directly in the line of sight can be problematic and even dangerous. At the same time, untrained people don’t like monocular displays outside the line of sight. It becomes (as Ed Tang said in the panel) a point of high friction to adoption.

Based on the many designs I have seen for AR glasses, we will see this all played out. Multiple companies are developing optical see-through AR glasses with monocular green MicroLEDs, color X-cube-based MicroLEDs, and LCOS-based displays with glass form-factor waveguide optics (both diffractive and reflective).

Introduction

The optics R&D  company Hypervision provided a detailed design analysis of the Apple Vision Pro’s optical design in June 2023 (see Apple Vision Pro (Part 4) – Hypervision Pancake Optics Analysis). Hypervision just released an interesting analysis exploring whether Micro-OLEDs, as used by the Apple Vision Pro, or LCDs used by Meta and most others, can support high 60 pixels per degree, angular resolution, and a wide FOV. Hypervision’s report is titled 60PPD: by fast LCD but not by micro OLED.

The optics R&D  company Hypervision provided a detailed design analysis of the Apple Vision Pro’s optical design in June 2023 (see Apple Vision Pro (Part 4) – Hypervision Pancake Optics Analysis). Hypervision just released an interesting analysis exploring whether Micro-OLEDs, as used by the Apple Vision Pro, or LCDs used by Meta and most others, can support high 60 pixels per degree, angular resolution, and a wide FOV. Hypervision’s report is titled 60PPD: by fast LCD but not by micro OLED. I’m going to touch on some highlights from Hypervision’s analysis. Please see their report for more details.

I Will Be at AWE Next Week

AWE is next week. I will be on the PANEL: Current State and Future Direction of AR Glasses at AWE on Wednesday, June 19th, from 11:30 AM to 12:25 PM. I still have a few time slots. If you want to meet, please email meet@kgontech.com.

AWE has moved to Long Beach, CA, south of LA, from its prior venue in Santa Clara. Last year at AWE, I presented Optical Versus Passthrough Mixed Reality, which is available on YouTube. This presentation was in anticipation of the Apple Vision Pro.

An AWE speaker discount code – SPKR24D- provides a 20% discount. You can register for AWE here.

Apple Vision Pro Sharpness Study at AWE 2024 – Need Help

As Hypervision’s analysis finds, plus reports I have received from users, the Apple Vision Pro’s sharpness varies from unit to unit. AWE 2024 is an opportunity to sample many Apple Vision Pro headsets to see how the focus varies from unit to unit. I will be there with my high-resolution camera.

While not absolutely necessary, it would be helpful if you could download my test pattern, located here, and install it on your Apple Vision Pro. If you want to help, contact me via meet@kgontech.com or flag me down at the show. I will be spending most of my time on the Expo floor. If you participate, you can remain anonymous or receive a mention of you or your company at the end of a related article thanking you for your participation. I can’t promise anything, but I thought it would be worth trying.

AVP Burry Image Controversy

My article Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3 was the first to report that the AVP was a little blurry. I compared high-resolution pictures showing the same FOV with the AVP and the Meta Quest 3 (MQ3) in that article.

This article caused controversy and was discussed in many forums and influencers, including Linus Tech Tips and Marquess Brownlee (see Apple Vision Pro—Influencing the Influencers & “Information Density” and “Controversy” of the AVP Being a Little Blurry Discussed on Marques Brownlee’s Podcast and Hugo Barra’s Blog).

I have recently been taking pictures through Bigscreen Beyond’s (BSB) headset and decided to compare it with the same test (above right). In terms of optical sharpness, it is between the AVP and the MQ3. Interestingly, the BSB headset has a slightly lower angular resolution (~32 pixels per degree) than the AVP (~40 ppd) in the optically best part of the lens where these crops were taken. Yet, the text and line patterns look better on the BSB than AVP.

Hypervision’s Correction – The AVP is Not Out of Focus, and the Optics are Blurry

I speculated that the AVP seemed out of focus in Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3. Hypervision corrected me that the softness could not be due to being out of focus. Hypervision has found that sharpness varies from one AVP to the next. The AVP’s best focus nominally occurs with an apparent focus of about 1 meter. Hypervision pointed out that if the headset’s device focus were slightly wrong, it would simply shift the apparent focus distance as the eye/camera would adjust to a small change in focus (unless it was so far off that eye/camera focusing was impossible). Thus, the blur is not a focus problem but rather a resolution problem with the optics.

Hypervision’s Analysis – Tolerances Required Beyond that of Today’s Plastic Optics

The AVP has very aggressive and complex pancake optics for a compact form factor while supporting a wide FOV with a relatively small Micro-OLED. Most other pancake optics have two elements, which mate with a flat surface for the polarizers and quarter waveplates that manipulate the polarized light to cause the light to pass through the optics twice (see Meta example below left). Apple has a more complex three-lens optic with curved polarizers and quarter waveplates (below right).

Based on my studies of how the AVP dynamically adjusts optical imperfections like chroma aberrations based on eye tracking, the AVP’s optics are “unstable” because, without dynamic correction, the imperfections would be seen as much worse.

Hypervision RMS Analysis

Hypervision did an RMS analysis comparing a larger LCD panel with a small Micro-OLED. It should probably come as no surprise that requiring about 1.8x (2.56/1.4) greater magnification makes everything more critical. The problem, as Hypervision points out, is that Micro-OLED on silicon can’t get bigger for many years due to semiconductor manufacturing limitations (reticle limit). Thus, the only way for Micro-OLED designs to support higher resolution and wider FOV is to make the pixels smaller and the optics much more difficult.

Hypervision Monte-Carlo Analysis

Hypervision then did a Monte-Carlo analysis factoring in optical tolerances. Remember, we are talking about fairly large plastic-molded lenses that must be reasonably priced, not something you would pay hundreds of dollars for in a large camera or microscope.

Hypervision’s 140 Degree FOV with 60PPD Approach

Hypervision believes that the only practical path to ~60PPD and ~140-degree FOV is with a 2.56″ LCD display. LCDs’ natural progression toward smaller pixels will enable higher resolution than their optics can support.

Conclusion

Overall, Hypervision makes a good case that current designs with Micro-OLED with pancake optics are already pushing the limits of reasonably priced optics. Using technology with somewhat bigger pixels makes resolving them easier, and having a bigger display makes supporting a wider FOV less challenging.

It might be that the AVP is slightly burry because it is already beyond the limits of a manufacturable design. So the natural question is, if AVP already has problems, how could they support higher resolution and wider FOV?

The size of Micro-OLEDs built on silicon backplanes is limited by a reticle limit of chip size of above ~1.4″ diagonally, at least without resorting to multiple reticle “stitching” (which is possible but not practical for a cost-effective device). Thus, for Micro-OLEDs to increase resolution, the pixels must be smaller, requiring even more magnification out of the optics. Then, increasing the FOV will require even more optical magnification of ever-tinier pixels.

LCDs have issues, particularly with black levels and contrast. Smaller illumination LEDs with local dimming may help, but they have not proven to work as well as micro-OLEDs.

Introduction

As I wrote in 2012’s Cynics Guide to CES—Glossary of Terms, when you see a demo at a conference, “sometimes you are seeing a “magic show” that has little relationship to real-world use.” I saw the Cogni Trax hard edge occlusion demo last week at SID Display Week 2024, and it epitomized the concept of being a “magic show.” I have been aware of Congi Trax for at least three years (and commented about the concept on Reddit), and I discovered they quoted me (I think a bit out of context) on its website (more on this later in the Appendix).

Cogni Trax has reportedly raised $7.1 million in 3 funding rounds over the last ~7 years, which I plan to show is unwarranted. I contacted Cogni Trax’s CEO (and former Apple optical designer on the Apple Vision Pro), Sajjad Khan, who was very generous in answering questions despite his knowing my skepticism about the concept.

Soft- Versus Hard-Edge Occlusion

Soft Edge Occlusion

In many ways, this article follows up on my 2021 Magic Leap 2 (Pt. 3): Soft Edge Occlusion, a Solution for Investors and Not Users, which detailed why putting an LCD in front of glass results in very “soft” occlusion.

Nobody will notice if you put a pixel-sized (angularly) dot on a person’s glasses. If it did, every dust particle on a person’s glasses would be noticeable and distracting. That is because a dot only a few millimeters from the eye is highly out of focus, and light rays from the real world will go around the dot before they are focused by the eye’s lens. That pixel dot will insignificantly dim several thousand pixels in the virtual image. As discussed in the Magic Leap soft occlusion article, the Magic Leap 2’s dimming pixel will cover ~2,100 pixels (angularly) in the virtual image and have a dimming effect on hundreds of thousands of pixels.

Hard Edge Occlusion (Optical and Camera Passthrough)

“Hard Edge Occlusion” means the precise, pixel-by-pixel light blocking. With camera passthrough AR (such as Apple Vision Pro), hard edge occlusion is trivial; one or more camera pixels are replaced by one or more pixels in the virtual image. Even though masking pixels is trivial with camera passthrough, there is still a non-trivial problem with getting the hard edge masking perfectly aligned to the real world. With passthrough mixel reality, the passthrough camera with its autofocus has focused the real world so it can be precisely masked.

With optical mixed reality hard edge occlusion, the real world must also be brought into focus before it can be precisely masked. Rather than going to a camera, the real world’s light goes to a reflective masking spatial light modular (SLM), typically LCOS, before combining it optically with the virtual image.

In Hard Edge (Pixel) Occlusion—Everyone Forgets About Focus, I discuss Arizona State University’s (ASU) optical solution for hard edge occlusion. Their solution has a set of optics that focuses the real world onto an SLM for masking. Then, a polarizing beam-splitting cube combines the result (with a change in polarization via two passes through a quarter waveplate not shown) after masking with a micro-display. While the ASU patent mentions using a polarizing beam splitter to combine the images, the patent fails to show or mention the need for a quarter waveplate between the SLM and beam splitter to work. One of the inventors, Hong Hua, was an ASU professor and a consultant to Magic Leap, and the patent was licensed to Magic Leap.

Other than being big and bulky, optically, what is wrong with the ASU’s hard edge occlusion includes:

• It only works to hard edge occlude at a distance set by the focusing. Ano
• The real world is “flatted” to be at the same focus as the virtual world.
• Polarization dims the real world by at least 50%. Additionally, viewing a polarized display device (like a typical LCD monitor or phone display) will be at least partially blocked by an amount that will vary with orientation relative to the optics.
• The real world is dimmed by at least 2x via the polarizing beam splitter.
• As the eye moves, the real world will move differently than it would with the eye looking directly. You are looking at the real world through two sets of optics with a much longer light path.

While Cogni Trax uses the same principle for masking the real world, it is configured differently and is much smaller and lighter. Both devices block a lot of light. Cogni Trax’s design blocks about 77% of the light, and they claim their next generation will block 50%. However, note that this is likely on top of any other light losses in the optical system.

Cogni Trax SID Display Week 2024 Demo

On the surface, the Cogni Trax demo makes it look like the concept works. The demo had a smartphone camera looking through the Cogni Trax optical device. If you look carefully, you will see that they block light from 4 areas of the real world (see arrow in the inset picture below), a Nike swoosh on top of the shoe, a QR code, the Coke in the bottle (with moving bubbles), and a partially darken the wall to the right to create a shadow of the bottle.

They don’t have a microdisplay with a virtual image; thus, they can only block or darken the real world and not replace anything. Since you are looking at the image on a cell phone and not with your own eyes, you have no sense of the loss of depth and parallax issues.

When I took the picture above, I was not planning on writing an article and missed capturing the whole setup. Fortunately, Robert Scoble put out an X-video that showed most of the rig used to align the masking to the real world. The rig supports aligning the camera and Cogni Trax device with six degrees of freedom. This demo will only work if all the objects in the scene are in a precise location relative to the camera/device. This is the epitome of a canned demo.

One could hand wave that developing SLAM, eye tracking, and 3-D scaling technology to eliminate the need for the rig is a “small matter of hardware and software” (to put it lightly). However, requiring a rig is not the biggest hidden trick in these demos; it is the basic optical concept and its limitations. The “device” shown (lower right inset) is only the LCOS device and part of the optics.

Cogni Trax Gen 1 Optics – How it works

Below is a figure of Congi Trax’s patent that will be used to diagram the light path. I have added some colorization to help you follow the diagram. The dashed-lined parts in the patent for combining the virtual image are not implemented in Cogni Trax’s current design.

The view of the real world follows a fairly torturous path. First, it goes through a polarizer where at least 50% of the light is lost (in theory, this polarizer is redundant due to the polarizing beam splitter to follow, but it is likely used to reduce any ghosting). It then bounces off of the polarizing beam splitter through a focusing element to bring the real world into focus on an LCOS SLM. The LCOS device will change the polarization of anything NOT masked so that on the return trip through the focusing element, it will pass through the polarizing beam splitter. The light then passes through the “relay optics,” then a Quarter Waveplate (QWP), off a mirror, and back through the quarter waveplate and relay optics. The two passes through the “relay optics” have to undo everything done to the light by the two passes through the focusing element. The two passes through the QWP will rotate the polarization of the light so that the light will bounce off the beam splitter and be directed at the eye via a cleanup polarizer. Optionally, as shown, the light can be combined with a virtual image from a microdisplay.

I find it hard to believe that real-world light will go through all that and will behave like nothing other than the light losses from polarization that have happened to it.

Cogni Trax provided a set of diagrams showing the light path of what they call “Alpha Pix.” I edited several of their diagrams together and added some annotations in red. As stated earlier, the current prototype does not have a microdisplay for providing a virtual image. If the virtual display device were implemented, its optics and combiner would be on top of everything else shown.

I don’t see this as a practical solution to hard-edge occlusion. While much less bulky than the ASU design, it still requires polarizing the incoming light and sending it through a torturous path that will further damage/distort real-world light. And this is before they deal with adding a virtual image. There is still the issue that the hard edge occlusion only works if everything being occluded is at approximately the same focus distance. If the virtual display is implemented, it would seem that the virtual image would need to be at approximately the same focus distance for it to be occluded correctly. Then, the hardware and software are required to get everything between the virtual and real world aligned with the eye. Even if the software and eye tracking were excellent, there where will still be a lag with any rapid head movement.

Cogni Trax Waveguide Design / Gen 2

Cogni Trax’s website and video discuss a “waveguide” solution for Gen 2. I found a patent (with excerpts right and below) from Cogni Trax for a waveguide approach to hard-edge occlusion that appears to agree with the diagrams in the video and on the website for their “waveguide.” I have outlined the path for the real world (in green) and the virtual image (in red).

Rather than using polarization, this method uses time-sequential modulation via a single Texas Instrument’s DLP/DMD. The DLP is used during part of the time block/pass light from the real world and as the virtual image display. I have included Figure 1(a), which gives the overall light path; Figures 1(c) and 1(d), which show the time multiplexing; Figure 6(a) with a front view of the design; and Figures 10 (a) and (b) which show a side view of the waveguide with the real world and virtual light paths respectively.

Other than not being polarized, the light follows a more torturous light path that includes a “fixed DMD” to correct for the micro-tilts of the real world by time-multiplexed displaying and masking DMD. In addition to all the problems I had with the Gen 1 design, I find putting the relatively small mirror (120 in Figure 1a) in the middle of the view very problematic as the view over or below the mirror will look very different than the view in the mirror with all the addiction optics. While it can theoretically give more light throughput and not require polarization of the real world, it can only do so by keeping the virtual display times short, which will mean more potential field sequential color breakup and lower color bit depth from the DLP.

Overall, I see Cogni Trax’s “waveguide” design as trading one set of problems for another set of probably worse image problems.

Conclusion

Perhaps my calling hard-edge occlusion a “Holy Grail” did not fully convey its impossibility. The more I have learned, examined, and observed this problem and its proposed solutions, the more clearly it seems impossible. Yes, someone can craft a demo that works for a tightly controlled setup with what is occluded at about the same distance, but it is a magic show.

The Cogni Trax demo is not a particularly good magic show, as it uses a massive 6-axis control rig to position a camera rather than letting the user put on a headset. Furthermore, the demo does not support a virtual display.

Cogni Trax’s promise of a future “waveguide” design appears to me to be at least as fundamentally flawed. According to the publicly available records, Cogni Trax has been trying to solve this problem for 7 years, and a highly contrived setup is the best they have demonstrated, at least publicly. This is more of a university lab project than something that should be developed commercially.

Based on his history with Apple and Texas Instruments, the CEO, Sajjad Khan, is capable, but I can’t understand why he is pursuing this fool’s errand. I don’t understand why over $7M has been invested, other than people blindly investing in former Apple designers without proper technical due diligence. I understand that high-risk, high-reward concepts can be worth some investment, but in my opinion, this does not fall into that category.

Appendix – Quoting Out of Context

Cogni Trax has quoted me in their video on their website as saying, “The Holy Grail of AR Displays.” It is not clear that A) I am referring to Hard Edge Occlusion (and not Cogni Trax) and B) I go on to say, “But it is likely impossible to solve for anything more than special cases of a single distance (flat) real world with optics.” The Audio in the Cogni Trax video from me, which is rather garbled, comes from a MARCH 30, 2021, AR Show, “KARL GUTTAG (KGONTECH) ON MAPPING AR DISPLAYS TO SUITABLE OPTICS (PART 2) at ~48:55 into the video (the occlusion issue is only briefly discussed).

Below, I have cited (with new highlighting in yellow) the section from my blog discussing hard edge occlusion from November 20, 2019, where Cogni Trax got my “Holy Grail” quote. This section of the article discusses the ASU design. This article discussed using a transmissive LCD for soft edge occlusion about 3 years before Magic Leap announced the Magic Leap 2 with such a method in July 2022.

Hard Edge (Pixel) Occlusion – Everyone Forgets About Focus

“Hard Edge Occlusion” is the concept of being able to block the real world with sharply defined edges, preferably to the pixel level. It is one of the “Holy Grails” of optical AR. Not having hard edge occlusion is why optical AR images are translucent. Hard Edge Occlusion is likely impossible to solve optically for all practical purposes. The critical thing most “solutions” miss (including US 20190324274) is that the mask itself must be in focus for it to sharply block light. Also, to properly block the real world, the focusing effect required depends on the distance of everything in the real world (i.e., it is infinitely complex).

The most common hard edge occlusion idea suggested is to put a transmissive LCD screen in the glasses to form “opacity pixels,” but this does not work. The fundamental problem is that the screen is so close to the eye that the light-blocking elements are out of focus. An individual opacity pixel will have a little darkening effect, with most of the light from a real-world point in space going around it and into the eye. A large group of opacity pixels will darken as a blurry blob.

Hard edge occlusion is trivial to do with pass-through AR by essentially substituting pixels. But it is likely impossible to solve for anything more than special cases of a single distance (flat) real world with optics. The difficulty of supporting even the flat-world special case is demonstrated by some researchers at the University of Arizona, now assigned to Magic Leap (the PDF at this link can be downloaded for free) shown below. Note all the optics required to bring the real world into focus onto “SLM2” (in the patent 9,547,174 figure) so it can mask the real world and solve the case for everything being masked being at roughly the same distance. None of this is even hinted at in the Apple application.

I also referred to hard edge occlusion as one of the “Holy Grails” of AR in a comment to a Magic Leap article in 2018 citing the ASU design and discussing some of the issues. Below is the comment, with added highlighting in yellow.

One of the “Holy Grails” of AR, is what is known as “hard edge occlusion” where you block light in-focus with the image. This is trivial to do with pass-through AR and next to impossible to do realistically with see-through optics. You can do special cases if all the real world is nearly flat. This is shown by some researchers at the University of Arizona with technology that is Licensed to Magic Leap (the PDF at this link can be downloaded for free: https://www.osapublishing.org/oe/abstract.cfm?uri=oe-25-24-30539#Abstract). What you see is a lot of bulky optics just to support a real world with the depth of a bookshelf (essentially everything in the real world is nearly flat).

FM: Magic Leap One – Instant Analysis in the Comment Section by Karl Guttag (KarlG) JANUARY 3, 2018 / 8:59 AM

Introduction

A notice in my LinkedIn feed mentioned that Brilliant Labs has started shipping its new Frame AR glasses. I briefly met with Brilliant CEO Bobak Tavangar at AWE 2023 (right) and got a short demonstration of its “Monocle” prototype. So, I investigated what Brilliant Labs was doing with its new “Frame.”

This started as a very short article, but as I put it together, I thought it would be an interesting example of making design decisions and trade-offs. So it became longer. Looking at the Frames more closely, I found issues that concerned me. I don’t mean to pick on Brillant Labs here. Any hardware device like the Frames is a massive effort, and they talk like they are concerned about their customers; I am only pointing out the complexities of supporting AI with AR for a wide audience.

While looking at how the Frame glasses work, I came across some information related to the Apple Vision Pro’s brightness (in nits), discussed last time in Apple Vision Pro Discussion Video by Karl Guttag and Jason McDowall. In the same way, the Apple Vision Pro’s brightness is being misstated as “5000 nits,” and the Brilliant Labs Frame’s brightness has been misreported as 3,000 nits. In both cases, the nits are the “potential” out of the display and not “to the eye” after the optics.

I’m also repeating the announcement that I will be at SID’s DisplayWeek next week and AWE next month. If you want to meet, please email meet@kgontech.com.

DisplayWeek (next week) and AWE (next month)

I will be at SID DisplayWeek in May and AWE in June. If you want to meet with me at either event, please email meet@kgontech.com. I usually spend most of my time on the exhibition floor where I can see the technology.

If you want to meet, please email meet@kgontech.com.

AWE has moved to Long Beach, CA, south of LA, from its prior venue in Santa Clara, and it is about one month later than last year. Last year at AWE, I presented Optical Versus Passthrough Mixed Reality, available on YouTube. This presentation was in anticipation of the Apple Vision Pro.

At AWE, I will be on the PANEL: Current State and Future Direction of AR Glasses on Wednesday, June 19th, from 11:30 AM to 12:25 PM.

There is an AWE speaker discount code – SPKR24D , which provides a 20% discount, and it can be combined with Early Bird pricing (which ends May 9th, 2024 – Today as I post this). You can register for AWE here.

Brilliant Labs Monocle & Frame “Simplistic” Optical Designs

Brillian Labs Monocle and Frame used the same basic optical architecture, but it is better hidden in the Frame design. I will start with the Monocle, as it is easier to see the elements and the light path. I was a little surprised that both designs use a very simplistic, non-polarized 50/50 beam splitter with its drawbacks.

Below (left) is a picture of the Monocle with the light path (in green). The Monocle (and Frame) both use a non-polarizing 50/50 beamsplitter. The splitter projects 50% of the display’s light forward and 50% downward to the (mostly) spherical mirror, magnifying the image and moving the apparent focus. After reflecting from the mirror, the light is split again in half, and ~25% of the light goes to the eye. The front project image will be mirrored, with an unmagnified view of the display that will be fairly bright. Front projection or “eye glow” is generally considered undesirable in social situations and is something most companies try to reduce/eliminate in their optical designs.

The middle picture above shows a picture I took of the Monocle from the outside, and you can see the light from the beam splitter projecting forward. Figures 5A and 6 (above right) from Brilliant Labs’ patent application illustrate the construction of the optics. The Monocle is made with two solid optical parts, with the bottom part forming part of the beam splitter and the bottom surface being shaped to form the curved mirror and then mirror coated. An issue with the 2-piece Monocle construction is that the beam splitter and mirror are below eye level, which requires the user to look down to see the image or position the whole device higher, which results in the user looking through the mirror.

The Frame optics work identically in function, but the size and spacing differ. The optics are formed with three parts, which enables Brilliant to position the beam splitter and mirror nearer the center of the user’s line of sight. But as Brilliant Lab’s documentation shows (right), the new Frame glasses still have the virtual (apparent) image below the line of sight.

Having the image below the line of sight reduces the distortion/artifacts of the real world by looking through the beam splitter when looking forward, but it does not eliminate all issues. The top seam of the beam splitter will likely be visible as an out-of-focus line.

The image below shows part of the construction process from a Brilliant Labs YouTube video. Note that the two parts that form the beamsplitter with its 50/50 semi-mirror coating have already been assembled to form the “Top.”

The picture above left is of a prototype taken by Forbes’ author Ben Sin of a Frame prototype from his article Frame Is The Most ‘Normal’ Looking AI Glasses I’ve Worn Yet. In this picture, the 50/50 beam splitter is evident.

Two Types of Birdbath

As discussed in Nreal Teardown: Part 1, Clones and Birdbath Basics and its Appendix: Second Type of Birdbath, there are two types of “birdbaths” used in AR. The Birdbath comprises a curved mirror (or semi-mirror) and a beamsplitter. It is called a “birdbath” because the light reflects out of the mirror. The beamsplitter can be polarized or unpolarized (more on this later). Birdbath elements are often buried in the design, such as the Lumus optical design (below left) with its curved mirror and beam splitter.

Many AR glasses today use the birdbath to change the focus and act as the combiner. The most common of these designs is where the user looks through a 50/50 birdbath mirror to see the real world (see Nreal/Xreal example below right). In this design, a polarised beam splitter is usually used with a quarter waveplate to “switch” the polarization after the reflection from the curved semi-mirror to cause the light to go through the beam splitter on its second pass (see Nreal Teardown: Part 1, Clones and Birdbath Basics for a more detailed explanation). This design is what I refer to as a “Look through the mirror” type of birdbath.

Brilliant Labs uses a “Look through the Beamsplitter” type of birdbath. Google Glass is perhaps the most famous product with this birdbath type (below left). This birdbath type has appeared in Samsung patents that were much discussed in the electronic trade press in 2019 (see my 2019 Samsung AR Design Patent—What’s Inside).

LCOS maker Raontech started showing a look through the beamsplitter reference design in 2018 (below right). The various segments of their optics are labeled below. This design uses a polarizing beam splitter and a quarter waveplate.

Brilliant Labs’ Thin Beam Splitter Causes View Issues

If you look at the RaonTech or Google Glass splitter, you should see that the beam splitter is the full height of the optics. However, in the case of the Frames and Monocle designs (right), the top and bottom beam splitter seams, the 50/50 mirror coating, and the curved mirror are in the middle of the optics and will be visible as out-of-focus blurs to the user.

Pros and Cons of Look-Through-Mirror versus Look-Through-Beamsplitter

The look-through-mirror birdbaths typically use a thin flat/plate beam splitter, and the curved semi-mirror is also thin and “encased in air.” This results in them being relatively light and inexpensive. They also don’t have to deal with the “birefringence” (polarization changing) issues associated with thick optical materials (particularly plastic). The big disadvantage of the look-through-mirror approach is that to see the real world, the user must look through both the beamsplitter and the 50/50 mirror; thus, the real world is dimmed by at least 75%.

The look-through-beamsplitter designs encase the entire design in either glass or plastic, with multiple glued-together surfaces coated or coated with films. The need to encase the design in a solid means the designs tend to be thicker and more expensive. Worse yet, typical injected mold plastics are birefringent and can’t be used with polarized optics (beamsplitters and quartwaveplates). Either heavy glass or higher-cost resin-molded plastics must be used with polarized elements. Supporting a wider FOV becomes increasingly difficult as a linear change in FOV results in a cubic increase in the volume of material (either plastic or glass) and, thus, the weight. Bigger optics are also more expensive to make. There are also optical problems when looking through very thick solid optics. You can see in the Raontech design above how thick the optics get to support a ~50-degree FOV. This approach “only” requires the user to look through the beam splitter, and thus the view of the real world is dimmed by 50% (or twice as much light gets through as the look-through-mirror method).

Pros and Cons Polarized Beam Splitter Birdbaths

Most companies with look-through-mirror and look-through-beamsplitter designs, but not Brilliant Labs, have gone with polarizing beam splitters and then use quarter waveplates to “switch” the polarization when the light reflects off the mirror. Either method requires the display’s light to make a reflective and transmissive pass via the beam splitter. With a non-polarized 50/50 beam splitter, this means multiplicative 50% losses or only 25% of the light getting through. With a polarized beam splitter, once the light is polarized with a 50% loss, with proper use of quarter waveplates, there are no more significant losses with the polarized beamsplitter.

Another advantage of the polarized optics approach is that front-projection can be mostly eliminated (there will be only a little due to scatter). The look-through-mirror method can be accomplished (as discussed in Nreal Teardown: Part 1, Clones and Birdbath Basics) with a second-quarter waveplate and a front polarizer. With the look-through-beamsplitter method, a polarizer before the beamsplitter will block the light that would project forward off the polarized beamsplitter.

As mentioned earlier, using polarized optics becomes much more difficult with the thicker solid optics associated with the look-through-beamsplitter method.

Brilliant Labs Frame Design Decision Options

It seems that at every turn in the decision process for the Frame and Monocle optics, Brilliant Labs chose the simplest and most economical design possible. By not using polarized optics, they gave up brightness and will have significant front projection. Still, they can use much less expensive injection-molded plastic optics that do not require polarizers and quart waveplates. They avoided using more expensive waveguides, which would be thinner but require LCOS or MicroLED (inorganic LED) projection engines, which may be heavier and larger. Although, the latest LCOS and MicroLED engines are getting to be pretty small and light, particularly for a >30-degree FOV (see DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8)).

Frames Brightness to the Eye – Likely >25% of 3,000 nits – Same Problem as Apple Vision Pro Reporting

As discussed in the last article on the Apple Vision Pro (AVP) in the Appendix: Rumor Mill’s 5,000 Nits Apple Vision Pro, reporters/authors constantly make erroneous comparisons of “display-out nits” with one device and to the nits-to-the-eye of other devices. Also, as stated last time, the companies appear to want this confusion by avoiding specifying the nits to the eye as they benefit from reporters and others using display device values.

I could not find an official Brilliant Labs value anywhere, but it seems to have told reporters that “the display is 3,000 nits,” which may not be a lie, but it is misleading. Most articles will dutifully give the “display number” but fail to say that they are “display device nits” and not what the user will see and leave it to the readers to make the mistake, while other reporters will make the error themselves.

Digitrends:

The display on Frame is monocular, meaning the text and graphics are displayed over the right eye only. It’s fairly bright (3,000 nits), though, so readability should be good even outdoors in sunlit areas.

Wearable:

As with the Brilliant Labs Monocle – the clip-on, open-source device that came before Frame – information is displayed in just one eye, with overlays being pumped out at around 3,000 nits brightness.

Android Central in androidcentral’s These AI glasses are being backed by the Pokemon Go CEO, who was at least making it clear that it was the display device numbers, but I still think most readers wouldn’t know what to do with this number. They added the tidbit that the panels were made by Sony, and they discussed pulse with modulation (also known as duty cycle). Interestingly, they talk about a short on-time duty cycle causing problems for people sensitive to flicker. In contrast, VR game fans favor a very short on-time duty cycle, what Brad Lynch of SadlyItsBradly refers to as low-persistence) to reduce blurring.

androidcentral’s These AI glasses are being backed by the Pokemon Go CEO

A 0.23-inch Sony MicroOLED display can be found inside one of the lenses, emitting 3,000 nits of brightness. Brilliant Labs tells me it doesn’t use PWM dimming on the display, either, meaning PWM-sensitive folks should have no trouble using it.

Below is a summary of Sony OLED Microdisplays aimed at the AR and VR market. On it, the 0.23 type device is listed with a max lumence of 3,000 nits. However, from the earlier analysis, we know that at most 25% of the light can get through Brilliant Labs Frame birdbath optics or at most 750 nits (likely less due to other optical losses). This number assumes that the device is driven at full brightness and that Brilliant Labs is not buying derated devices at a lower price.

I can’t blame Brilliant Labs because almost every company does the same in terms of hiding the ball on to-the-eye brightness. Only companies with comparatively high nits-to-the-eye values (such as Lumus) publish this spec.

Sony Specifications related to the Apple Vision Pro

The Sony specifications list a 3.5K by 4 K device. The industry common understanding is that Apple designed a custom backplane for the AVP but then used Sony’s OLED process. Notice the spec of 1,000 cd/m2 (candelas per meter squared = nits) at a 20% duty ratio. While favorable for VR gamers wanting less motion blur, the low on-duty cycle time is also a lifetime issue. The display device probably can’t handle the heat from being driven for a high percentage of the time.

It would be reasonable to assume that Apple is similarly restricted to about a 20% on-duty cycle. As I reported last time in the Apple Vision Pro Discussion Video by Karl Guttag and Jason McDowall, I have measured the on-duty cycle of the AVP to be about 18.4% or close to Sony’s 20% for their own device.

The 5,000 nits cited by MIT Tech Review are the raw displays before the optics, whereas the nits for the MQ2 were those going to the eye. The AVP’s (and all other) pancake optics transmit about 11% (or less) of the light from an OLED in the center. With Pancake optics, there is the polarization of the OLED (>50% loss), a transmissive pass, and a reflective pass through a 50/50 mirror, which starts with at most 12.5% (50% cubed) before considering all the other losses from the optics. Then, there is the on-time-duty cycle of the AVP, which I have measured to be about 18.4%. VR devices want the on-time duty cycle to be low to reduce motion blur with the rapid motion of the head and 3-D game. The MQ3 only has a 10.3% on-time duty cycle (shorter duty cycles are easier with LED-illuminated LCDs). So, while the AVP display devices likely can emit about 5,000 nits, the nits reaching the eye are approximately 5,000 nits x 11% x 18.4% = 100 nits.

View Into the Frame Glasses

I don’t want to say that Brilliant Labs is doing anything wrong or that other companies don’t often do the same. Companies often take pictures and videos of new products using non-functional prototypes because the working versions aren’t ready when shooting or because they look better on camera. Still, I want to point out something I noticed with the pictures of the CEO, Bobak Tavangar (right), that was published in many of the articles in the Frames glasses. I didn’t see the curved mirror and the 50/50 beam splitter.

In a high-resolution version of the picture, I could see the split in the optics (below left) but not the darkened rectangle of the 50/50 mirror. So far, I have found only one picture of someone wearing the Frame glasses from Bobak Tavangar’s post on X. It is of a person wearing what appears to be a functional Frame in a clear prototype body (below right). In the dotted line box, you can see the dark rectangle from the 50/50 mirror and a glint from the bottom curved mirror.

I don’t think Brilliant Labs is trying to hide anything, as I can find several pictures that appear to be functional frames, such as the picture from another Tavangar post on X showing trays full of Frame devices being produced (right) or the Forbes picture (earlier in the Optical section).

What was I hoping to show?

I’m trying to show what the Frame looks like when worn to get an idea of the social impact of wearing the glasses. I was looking for a video of someone wearing them with the Frame turned on, but unfortunately, none have surfaced. From the design analysis above, I know they will project a small but bright image view with a mirror image of the display off of the 50/50 mirror, but I have not found an image showing the working device from the outside looking in.

Exploded View of the Frame Glasses

The figure below is taken from Brilliant Lab’s online manual for the Frame glasses (I edited it to reduce space and inverted the image to make it easier to view). By AR glasses standards, the Frame design is about as simple as possible. The choice of two nose bridge inserts is not shown in the figure below.

There is only one size of glasses, which Brilliant Labs described in their AMA as being between a “medium and large” type frame. They say that the temples are flexible to accommodate many head widths. Because the Frames are monocular, IPD is not the problem it would be with a biocular headset.

AddOptics is making custom prescription lenses for the Frames glasses

Brilliant Labs is partnering with AddOptics to make prescription lenses that can be ‘Precision Bonded’ to Frames using a unique optical lens casting process. For more on AddOptics, see CES 2023 (Part 3) – AddOptics Custom Optics and my short follow-up in Mixed Reality at CES & AR/VR/MR 2024 (Part 2 Mostly Optics).

Bonding to the Frames will make for a cleaner and more compact solution than the more common insert solution, but it will likely be permanent and thus a problem for people whose prescriptions change. In their YouTube AMA, Brilliant Labs said they are working with AddOptics to increase the range of prescription values and support for astigmatism.

They didn’t say anything about bifocal or progressive lens support, which is even more complicated (and may require post-mold grinding). As the virtual image is below the centerline of vision, it would typically be where bifocal and progressive lenses would be designed for reading distance (near vision). In contrast, most AR and VR glasses aim to put the virtual image at 2 meters, considered “far vision.”

The Frame’s basic specs

Below, I have collected the basic specs on the Frame glasses and added my estimate for the nits to the eye. Also shown below is their somewhat comical charging adapter (“Mister Charger”). None of these specs are out of the ordinary and are generally at the low end for the display and camera.

• Monocular 640×400 resolution OLED Microdisplay
• ~750nits to the eye (based on reports of a 3,000 Sony Micro-OLED display device)
  • (90% on-time duty cycle using an
• 20-Degree FOV
• Weight ~40 grams
• 1280×720 camera
• Microphone
• 6 axis IMU
• Battery 222mAh  (plus 149mAh top-up from charging adapter)
  • With 80mA typical power consumption when operating 0.580 on standby)
• CPU nRF52840 Cortex M4F (Nordic ARM)
• Bluetooth 5.3

Everything in AR Today is “AI”

Brilliant Labs is marketing the frames as “AI Glasses.” The “AI” comes from Brilliant Lab’s Noa ChatGPT client application running on a smartphone. Brillant Labs says the hardware is “open source” and can be used by other companies’ applications.

I’m assuming the “AI” primarily runs on the Noa cell phone application, which then connects to the cloud for the heavy-lifting AI. According to their video by Brillant Labs, while on the Monocle, the CPU only controls the display and peripherals, they plan to move some processing onto the Frame’s more capable CPU. Like other “AI” wearables, I expect simple questions will get immediate responses while complex questions will wait on the cloud.

Conclusions

To be fair, designing glasses and wearable AR products for the mass market is difficult. I didn’t intend to pick on Brilliant Lab’s Frames; instead, I am using it as an example.

With a monocular, 20-degree FOV below the center of the person’s view, the Frames are a “data snacking” type AR device. It is going to be competing with products like the Human AI projector (which is a joke — see: Humane AI – Pico Laser Projection – $230M AI Twist on an Old Scam), the Rabbit R1, Meta’s (display-less) Ray Ban Wayfarer, other “AI” audio glasses, and many AR-AI glasses similar to the Frame that are in development.

This blog normally concentrates on display and optics, and on this score, the Frame’s optics are a “minimal effort” to support low cost and weight. As such, they have a lot of problems, including:

• Small 20-degree FOV that is set below the eyes and not centered (unless you are lucky with the right IPD)
• Due to the way the beam 50/50 splitter cuts through the optics, it will have a visible seam. I don’t think this will be pleasant to look through when the display is off (but I have not tried them yet). You could argue that you only put them on “when you need them,” but that negates most use cases.
• The support for vision correction appears to lock the glasses to a single (current) prescription.
• Regardless of flexibility, the single-size frame will make the glasses unwearable for many people.
• The brightness to the eye of probably less than 750 nits is not bright enough for general outdoor use in daylight. It might be marginal if used combined with clip-on sunglasses or if they are used in the shade.

As a consumer, I hate the charger adapter concept. Why they couldn’t just put a USB-C connector on the glasses is beyond me and a friction point for every user. Users typically have dozens of USB-C power cables today, but your device is dead if you forget or lose the adaptor. Since these are supposed to be prescription glasses, the idea of needing to take them off to charge them is also problematic.

While I can see the future use model for AI prescription glasses, I think a display, even one with a small FOV, will add significant value. I think Brillant Labs’s Frames are for early adopters who will accept many faults and difficulties. At least they are reasonably priced at $349, by today’s standards, and don’t require a subscription for basic services without too many complex AI queries requiring the cloud.

Introduction

As discussed in Mixed Reality at CES and the AR/VR/MR 2024 Video (Part 1 – Headset Companies), Jason McDowall of The AR Show recorded over four hours of video discussing the 50 companies I met at CES and AR/VR/MR. The last thing we discussed for about 50 minutes was the Apple Vision Pro (AVP).

The AVP video amounts to a recap of the many articles I have written on the AVP. Where appropriate, I will give links to my more detailed coverage in prior articles and updates rather than rehash that information in this article.

It should be noted that Jason and I recorded the video on March 25th, 2024. Since then, there have been many articles from tech magazines saying the AVP sales are lagging, often citing Bloomberg’s Mark Gurman’s “Demand for demos is down” and Analyst Ming Quo reporting, “Apple has cut its 2024 Vision Pro shipments to 400–450k units (vs. market consensus of 700–800k units or more).” While many reviewers cite the price of the AVP, I have contended that price was not the problem as it was in line with a new high-tech device (adjusted for inflation, it is about the same price as the first Apple II). My criticism focuses on the utility and human factors. In high-tech, the cost is usually a fixable problem with time and effort, and people will pay more if something is of great utility.

I said the Apple Vision Pro would have utility problems before it was announced. See my 2023 AWE Presentation “Optical Versus Passthrough Mixed Reality“) and my articles on the AVP. I’m not about bashing a product or concept; when I find faults, I point them out and show my homework, so to speak, on this blog and in my presentations.

Before the main article, I want to repeat the announcement that I plan to go to DisplayWeek in May and AWE in June. I have also included a short section on YouTube personality/influence Marques Browlee’s Waveform Podast and Hugo Barra’s (former Head of Oculus at Meta) blog article discussing my controversial (but correct) assessment that the Apple Vision Pro’s optics are slightly out of focus/blurry.

DisplayWeek and AWE

I will be at SID DisplayWeek in May and AWE in June. If you want to meet with me at either event, please email meet@kgontech.com. I usually spend most of my time on the exhibition floor where I can see the technology.

AWE has moved to Long Beach, CA, south of LA, from its prior venue in Santa Clara, and it is about one month later than last year. Last year at AWE, I presented Optical Versus Passthrough Mixed Reality, available on YouTube. This presentation was in anticipation of the Apple Vision Pro.

At AWE, I will be on the PANEL: Current State and Future Direction of AR Glasses on Wednesday, June 19th, from 11:30 AM to 12:25 PM with the following panelists:

• Jason McDowall – The AR Show (Moderator)
• Jeri Ellsworth – Tilt Five
• Adi Robertson – The Verge
• Edward Tang – Avegant
• Karl M Guttag – KGOnTech

There is an AWE speaker discount code – SPKR24D , which provides a 20% discount, and it can be combined with Early Bird pricing (which ends May 9th, 2024). You can register for AWE here.

“Controversy” of the AVP Being a Little Blurry Discussed on Marques Brownlee’s Podcast and Hugo Barra’s Blog

As discussed in Apple Vision Pro – Influencing the Influencers & “Information Density,” which included citing this blog on Linus Tips, this blog is read by other influencers, media, analysts, and key people at AR/VR/MR tech companies.

Marques Brownlee (MKBHD), another major YouTube personality, Waveform Podcast/WVFRM YouTube channel, discussed (link to the YouTube discussion) my March 1st article on Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3. Marques discussed Hugo Barra’s (former Head of Oculus at Meta) blog’s March 11, 2024 “Hot Take” article (about 1/3rd of the way down) on my blog article.

According to MKBHD and Hugo Barra, my comments about Vision Pro are controversial, but they agree that it would make sense based on my evidence and their experience. My discussion with Jason was recorded before the Waveform Podcast came out. I’m happy to defend and debate this issue.

Outline of the Video and Additional Information

The Video The times in blue on the left of each subsection give the link to the YouTube video section discussing that subject.

00:16 Ergonomics and Human Factors

I wrote about the issues with the AVP’s human factors design in Apple Vision Pro (Part 2) – Hardware Issues Mechanical Ergonomics. In a later article in CES Part 2, I compared the AVP to the new Sony XR headset in the Sony XR (and others compared to Apple Vision Pro) section.

08:23 Lynx and Hypervision

I wrote the article comparing the new Sony XR headset to the AVP mentioned the Lynx R1, first shown in 2021, in this comparison. But I didn’t realize how much they were alike until I saw a post somewhere (I couldn’t find it again) by Lynx’s CEO, Stan Larroque saying how much they were alike. It could be a matter of form following function, but how much they are alike from just about any angle is rather striking.

While on the subject of Lynx and Apple. Lynx used optic by Limbak for the Lynx R1. As I broke in December 2022 Limbak Bought by “Large US Company” (which soon was revealed as Apple) and discussed in more detail in a 2022 Video with Brad Lynch, I don’t like the R1’s Limbak “catadioptric” (combined mirror and refractive) optics. While the R1 optics are relatively thin, like pancake optics, they cause a significant loss of resolution due to their severe distortion, and worse, they have an optical discontinuity in the center of the image unless the eye is perfectly aligned.

In May 2023, Lynx and Hypervision announced that they were working together. In Apple Vision Pro (Part 4)—Hypervision Pancake Optics Analysis, Hypervision detailed the optics of the Apple Vision Pro. That article also discusses the Hypervision pancake optics it was showing at AR/VR/MR 2023. Hypervision demonstrated single pancake optics with a 140-degree FOV (the AVP is about 90 degrees) and blended dual pancake optics with a 240-degree FOV (see below right).

10:59 Big Screen Beyond Compared to AVP Comfort Issues

When I was at the LA SID One Day conference, I stopped by Big Screen Beyond to try out their headset. I wore Big Screen’s headset for over 2 hours and didn’t have any of the discomfort issues I had with the AVP. With the AVP, my eyes start bothering me after about 1/2 hours and are pretty sore by 1 hour. There are likely two major factors: one is that the AVP is applying pressure to the forehead, and the other is that something is not working right optically with the AVP.

Big Screen Beyond has a silicon gel-like custom interface that is 3-D printed based on a smartphone face scan. Like the AVP, they have magnetic prescription inserts. While the Big Screen Beyond was much more comfortable, the face interface has a large contact area with the face. While not that uncomfortable, I would like something that breathed more. When you remove the headset, you can feel the preparation evaporating from where the interface was contacting your face. I can’t imagine anyone wearing makeup being happy (the same with the with the AVP or any headset that presses against the face).

On a side note, I was impressed by Big Screen Beyond’s statement that it is cash flow positive. It is a sign that they are not wildly spending money on frills and that they understand the market they are serving. They are focused on serving dedicated VR gamers who want to connect the headset to a powerful computer.

Related to the Big Screen Beyond interface, a tip I picked up on Reddit is that you can use a silicon face pad made for the Meta Quest 2 or 3 on the AVP’s face interface (see above right). The silicon face pad gives some grip to the face interface and reduces the pressure required to hold the AVP steady. The pad adds about 1mm, but it so happens that I had recently swapped my original AVP face interface for one that is 5mm shorter. Now, I barely need to tighten the headband. A downside to the silicon pad, like the Big Screen Beyond, is that it more or less forms a seal with your face, and you can feel the perspiration evaporating when you remove it.

13:16 Some Basic AVP Information

In the video, I provide some random information about the AVP. I wanted to go into detail here about the often misquoted brightness of the AVP.

I started by saying that I have read or watched many people state that the AVP is much brighter than the Meta Quest 3 (MQ3) or Meta Quest Pro (MQP). They are giving ridiculously high brightness/nits values for the AVP. As I reported in my March 7th, 2024, comments in the article Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3, the AVP outputs to the eye about 100 nits and is only about 5-10% brighter than the MQ3 and ~20% less than the MQP.

I will explain how this came about in the Appendix at the end. And to this day, if you do a Google search (captured below), it will prominently state that the AVP has a “50-fold improvement over the Meta’s Quest 2, which hits just 100 nits,” citing MIT Technology Review.

Nits are tricky to measure in a headset without the right equipment, and even then, they vary considerably from the center (usually the highest to the periphery).

The 5,000 nits cited by MIT Tech Review are the raw displays before the optics, whereas the nits for the MQ2 were those going to the eye. The AVP’s (and all other) pancake optics transmit about 11% (or less) of the light from an OLED in the center. With Pancake optics, there is the polarization of the OLED (>50% loss), a transmissive pass, and a reflective pass through a 50/50 mirror, which starts with at most 12.5% (50% cubed) before considering all the other losses from the optics. Then, there is the on-time-duty cycle of the AVP, which I have measured to be about 18.4%. VR devices want the on-time duty cycle to be low to reduce motion blur with the rapid motion of the head and 3-D game. The MQ3 only has a 10.3% on-time duty cycle (shorter duty cycles are easier with LED-illuminated LCDs). So, while the AVP display devices likely can emit about 5,000 nits, the nits reaching the eye are approximately 5,000 nits x 11% x 18.4% = 100 nits.

18:59 Computer Monitor Replacement is Rediculous

I wrote a three-part series on why I think monitor replacement by the Apple Vision Pro is ridiculous. Please see Apple Vision Pro (Part 5A) – Why Monitor Replacement is Ridiculous, Part 5B, and Part 5C. There are multiple fundamental problems that neither Apple nor anyone else is close to solving. The slide on the right summarizes some of the big issues.

Nyquist Sampling – Resampling Causes Blurring & Artifacts

I tried to explain the problem in two ways, one based on the frequency domain and the other on the spatial (pixel) domain.

19:29 Frequency Domain Discussion

Anyone familiar with signal processing may remember that a square wave has infinite odd harmonics. Images can be treated like 2-dimensional signals. A series of equally spaced, equal-width horizontal lines looks like a square wave in the vertical dimension. Thus, to represent them perfectly with a 3-D transform requires infinite resolution. Since the resolution of the AVP (or any VR headset) is limited, there will be artifacts such as blurring, wiggling, and scintillation.

As I pointed out in (Part 5A), computers tend to “cheat” and distort text and graphics to fit on the pixel grid and thus sidestep the Nyquist sampling problem that any VR headset must face when trying to make a 2-D image appear still in 3-D space. Those who know signal processing know that the Nyquist rate is 2x the highest frequency component. However, as noted above, horizontal lines have infinite frequency. Hence, some degradation is inevitable, but then we only have to beat the resolution limit of the eye, which, in effect, acts as a low-pass filter. Unfortunately, the AVP’s display is about 2-3x too low linearly (4-9x in two dimensions) in resolution for the artifacts not to be seen by a person with good vision.

22:15 Spatial Domain Discussion

To avoid relying on signal processing theory, in (Part 5A), I gave the example of how a single display pixel can be translated into 3-D space (right). The problem is that a pixel the size of a physical pixel in the headset will always cover parts of four physical pixels. Worse yet, with the slightest movement of a person’s head, how much of each pixel and even which pixels will be constantly changing, causing temporal artifacts such as wiggling and scintillation. The only way to reduce the temporal artifacts is to soften (low pass filter) the image in the resampling process.

23:19 Optics Distortion

In addition to the issues with representing a 2-D image in 3-D space, the AVP’s optics are highly distorting, as discussed in Apple Vision Pro’s (AVP) Image Quality Issues—First Impressions. The optical distortions can be “digitally corrected” but face the same resample issues discussed above.

25:51 Close-Up Center Crop and Foveated Boundary

The figures shown in this part of the video come from Apple Vision Pro’s (AVP) Image Quality Issues – First Impressions, and I will refer you to that article rather than repeat it here.

28:52 AVP’s Pancake Optics and Comparison to MQ3 and Birdbath

Much of this part of the video is covered in more detail in Apple Vision Pro’s (AVP) Image Quality Issues—First Impressions.

Using Eye Tracking for Optics Has Wider Implications

A key point made in the video is that the AVP’s optics are much more “aggressive” than Meta’s, and as a result, they appear to require dynamic eye tracking to work well. I referred to the AVP optics as being “unstable.” The AVP is constantly pre-correcting for distortion and color based on eye tracking. While the use of eye tracking for Foveated Rendering and control input is much discussed by Apple and others, using eye tracking to correct the optics has much more significant implications, which may be why the AVP has to be “locked” onto a person’s face.

Eye tracking for foveated rendering does not have to be nearly as precise as it is for correction, but using it for optical correction does. This leads me to speculate that the AVP requires the facial interfaces to lock the headset to the face, which is horrible regarding human factors, to support pre-correcting the optics. This follows my rule, “when smart people do something that appears dumb, it is because the alternative was worse.”

Comparison to (Nreal/Xreal) Birdbath

One part not discussed in the video or that article but shown in the associated figure (below) is the similarity of Pancake Optics are similar to Birdbath Optics. Nreal (now Xreal) Birdbath optics are discussed in my Nreal teardown series in Nreal Birdbath Overview.

Both pancake and birdbath optics start by polarizing the image from an OLED microdisplay. They use quarter waveplates to “switch” the polarization, causing it to bounce off a polarizer and then pass through it. They both use a 50/50 coated semi-mirror. They both use a combination of refractive (lens) and reflective (mirror) optics. In the case of the birdbath, the polarizer acts as a beam splitter to the OLED display so it does not block the view out, whereas with pancake optics, everything is inline.

31:34 AVP Color Uniformity Problem

The color uniformity and the fact that the color shift moves around with eye movement were discussed in Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3.

32:11 Comparing Resolution vs a Monitor

In Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3, I compared the resolution of the AVP (below left) to various computer monitors (below right) and the Meta Quest 3.

Below is a close-up crop of the center of the same image shown on the AVP, a 28″ monitor, and the Meta Quest 3. See the article for an in-depth explanation.

33:03 Vision OS 1.1 Change in MacBook mirror processing

I received and saw some comments about my Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3 that Vision OS 1.1 MacBook mirroring was sharper. I had just run a side-by-side comparison of displaying an image from a file on the AVP versus displaying the same image via mirroring a MacBook in Apple Vision Pro Displays the Same Image Differently Depending on the Application. So, I downloaded Vision OS 1.1 to the AVP and reran the same test, and I found a clear difference in the rendering of the MacBook mirroring (but not the display from the AVP file). However, it was not that the MacBook mirror image was shaper per se, but it was less bold. Even in the thumbnails below (click on them to see the full-size images). In the thumbnails below, note how the text looks less bold on the right side of the left image (OS 1.2) versus the right side of the right image.

Below are crops from the two images above, with the OS 1.1 image on the top and OS 1.0 on the bottom. The MacBook mirroring comes from the right sides of both images. Note how much bold the text and lines are in the OS 1.1 crop.

35:57 AVP Passthrough Cameras in the Wrong Location

38:43 AVP’s Optics are Soft/Blurry

As stated in Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3, the AVP optics are a little soft. According to Marquees Brownlee (see above) and others, my statement has caused controversy. I have heard others question my methods, but I have yet to see any evidence to the contrary.

I have provided my photographic evidence (right) and have seen it with my eyes by swapping headsets back and forth with high-resolution content. For comparison, the same image was displayed on the Meta Quest 3, and the MQ3 was clearly sharper. The “blur” on the AVP is similar to what one would see with a Gaussian blur with a radius of about 0.5 to 1 pixel.

Please don’t confuse “pixel resolution” with optical sharpness. The AVP has more pixels per degree, but the optics are a bit out of focus and, thus, a little blurry/soft. One theory is that it is being done to reduce the screen door effect (seeing the individual pixels) and make the images on the AVP look smoother.

The slight blurring of the AVP may reduce the screen door effect as the gap between pixels is thinner on the OLED displays than on the MQ3’s LCDs. But jaggies and scintillation are still very visible on the AVP.

41:41 Closing Discussion: “Did Apple Move the Needle?”

The video wraps up with Jason asking the open-ended question, “Did Apple Move the Needle?” I discuss whether it will replace a cell phone, home monitor(s), laptop on the road, or home TV. I think you can guess that I am more than skeptical that the AVP now or in the future will change things for more than a very small fraction of the people who use cell phones, laptops, and TVs. As I say about some conference demos, “Not everything that would make a great theme park experience is something you will ever want in your home to use regularly.”

Appendix: Rumor Mill’s 5,000 Nits Apple Vision Pro

When I searched the Internet to see if anyone had independently reported on the brightness of the AVP, I got the Google search answer in big, bold letters: “5,000 Nits” (right). Then, I went to the source of this answer, and it was none other than the MIT Technology Review. I then thought they must be quoting the display’s brightness, not the headset’s, but it reports that it is a “50-fold improvement over Meta Quest 2,” which is ridiculous.

I see this all the time when companies quote a spec for the display device, and it gets reported as the headset’s brightness/nits to the eye. The companies are a big part of the problem because most headset makers won’t give a number for the eye’s brightness in their specs. I should note that with almost all headset optics, the peak nits in the center will be much higher than those in the periphery. Through the years, one thing I have found that all companies exaggerate in their marketing is the brightness, either in lumens for projectors or nits for headsets.

An LCOS or DLP display engine can output over a million nits into a waveguide, but that number is so big (almost never given) that it is not confused with the nits to the eye. Nits are a function of light output (measured in Lumens) and the ability to collimate the light (a function of the size of the light source and illumination optics).

The “5,000 nits” source was a tweet by Ross Young of DSCC. Part of the Tweet/X thread is copied on the right. A few respondents understood this could not be the nits to the eye, and a few responders understood that it could not be to the eye. Responder BattleZxeVR even got the part about the duty cycle being a factor, but that didn’t stop many other later responders from getting it wrong.

Citing some other publications that didn’t seem to understand the difference between nits-in versus nits-out:

Quoting from The Daejeon Chronicles (June 2023): Apple Vision Pro Screens: 5,000 Nits of Wholesome HDR Goodness (with my bold emphasis):

Dagogo Altraide of ColdFusion has this to say about the device’s brightness capability:

“The screens have 5,000 nits of peak brightness, and that’s a lot. The Meta Quest 2, for example, maxes out at about 100 nits of brightness and Sony’s PS VR, about 265 nits. So, 5,000 nits is crazy. According to display analyst Ross Young, this 5,000 nits of peak brightness isn’t going to blind users, but rather provide superior contrast, brighter colors and better highlights than any of the other displays out there today.”

Quoting from Mac Rumors (May 2023): Apple’s AR/VR Headset Display Specs: 5000+ Nits Brightness for HDR, 1.41-Inch Diagonal Display and More:

With ~5000 nits brightness or more, the AR/VR headset from Apple would support HDR or high dynamic range content, which is not typical for current VR headsets on the market. The Meta Quest 2, for example, maxes out around 100 nits of brightness and it does not offer HDR, and the HoloLens 2 offers 500 nits brightness. Sony’s PSVR 2 headset has around 265 nits of brightness, and it does have an advertised HDR feature when connected to an HDR display.

The flatpanelshd (June 2023): Apple Vision Pro: Micro-OLEDs with 3800×3000 pixels & 90/96Hz – a paradigm shift did understand that the 5,000 nist was the display device and not to the eye:

DSCC has previously said that the micro-OLED displays deliver over 5000 nits of brightness but a good portion of that is typically lost due to the lenses and the display driving method.

As I wrote in Apple Vision Pro (Part 1) – What Apple Got Right Compared to The Meta Quest Pro, Snazzy Labs had an excellent explanation of the issues with the applications shown by Apple at the AVP announcement (it is a fun and informative video). But in another otherwise excellent video, What Reviewers Aren’t Telling You About Apple Vision Pro, I have to give him credit for recognizing that the MIT Tech Review had confabulated the display’s brightness with the headset’s brightness. But then hazarded a guess that it would be “after the optics, I bet it’s around 1,000 nits.” His guess was “just a bit outside” by about 10x. I do not want to pick on Snazzy Labs, as I love the videos I have seen from them, but I want to point out how much even technically knowledgeable people without a background in optics underestimate the light losses in headset optics.

Update 2/21/22: I added a discussion of the DLP’s new frame rates and its potential to address field sequential color breakup.

Introduction

In part 3 of my combined CES and AR/VR/MR 2024 coverage of over 50 Mixed Reality companies, I will discuss display companies.

As discussed in Mixed Reality at CES and the AR/VR/MR 2024 Video (Part 1 – Headset Companies), Jason McDowall of The AR Show recorded more than four hours of video on the 50 companies. In editing the videos, I felt the need to add more information on the companies. So, I decided to release each video in sections with a companion blog article with added information.

Outline of the Video and Additional Information

The part of the video on display companies is only about 14 minutes long, but with my background working in displays, I had more to write about each company. The times in blue on the left of each subsection below link to the YouTube video section discussing a given company.

00:10 Lighting Silicon (Formerly Kopin Micro-OLED)

Lighting Silicon is a spinoff of Kopin’s micro-OLED development. Kopin started making micro-LCD microdisplays with its transmissive color filter “Lift-off LCOS” process in 1990. 2011 Kopin acquired Forth Dimension Displays (FDD), a high-resolution Ferroelectric (reflective) LCOS maker. In 2016, I first reported on Kopin Entering the OLED Microdisplay Market. Lighting Silicon (as Kopin) was the first company to promote the combination of all plastic pancake optics with micro-OLEDs (now used in the Apple Vision Pro). Panasonic picked up the Lighting/Kopin OLED with pancake optics design for their Shift All headset (see also: Pancake Optics Kopin/Panasonic).

At CES 2024, I was invited by Chris Chinnock of Insight Media to be on a panel at Lighting Silicon’s reception. The panel’s title was “Finding the Path to a Consumer-Friendly Vision Pro Headset” (video link – remember this was made before the Apple Vision Pro was available). The panel started with Lighting Silicon’s Chairman, John Fan, explaining Lighting Silicon and its relationship with Lakeside Lighting Semiconductor. Essentially, Lightning Semiconductor designs the semiconductor backplane, and Lakeside Lighting does the OLED assembly (including applying the OLED material a wafer at a time, sealing the display, singulating the displays, and bonding). Currently, Lakeside Lighting is only processing 8-inch/200mm wafers, limiting Lighting Silicon to making ~2.5K resolution devices. To make ~4K devices, Lighting Semiconductor needs a more advanced semiconductor process that is only available in more modern 12-inch/300mm FABs. Lakeside is now building a manufacturing facility that can handle 12-inch OLED wafer assembly, enabling Lighting Silicon to offer ~4K devices.

Related info on Kopin’s history in microdisplays and micro-OLEDs:

• 2022 AWE Video Discussion with Brad Lynch Kopin (LCOS and OLED microdisplays)
• 2021 Pancake Optics Kopin/Panasonic
• 2013 Kopin Displays and Near Eye (Follow-Up to Seeking Alpha Article)
• 2013 Extended Temperature Range with LC-Based Microdisplays (about Kopin)

02:55 RaonTech

RaonTech seems to be one of the most popular LCOS makers, as I see their devices being used in many new designs/prototypes. Himax (Google Glass, Hololens 1, and many others) and Omnivision (Magic Leap 1&2 and other designs) are also LCOS makers I know are in multiple designs, but I didn’t see them at CES or the AR/VR/MR. I first reported on RaonTech at CES 2018 (Part 1 – AR Overview). RaonTech makes various LCOS devices with different pixel sizes and resolutions. More recently, they have developed a 2.15-micron pixel pitch field sequential color pixel with an “embedded spatial interpolation is done by pixel circuit itself,” so (as I understand it) the 4K image is based on 2K data being sent and interpolated by the display.

In addition to LCOS, RaonTech has been designing backplanes for other companies making micro-OLED and MicroLED microdisplays.

04:01 May Display (LCOS)

May Display is a Korean LCOS company that I first saw at CES 2022. It surprised me, as I thought I knew most of the LCOS makers. May is still a bit of an enigma. They make a range of LCOS panels, their most advanced being an 8K (7980 x 4,320) 3.2-micron pixel pitch. May also makes a 4K VR headset with a 75-degree FOV using their LCOS devices.

May has its own in-house LCOS manufacturing capability. May demonstrated using its LCOS devices in projectors and VR headsets and showed them being used in a (true) holographic projector (I think using phase LCOS).

May Display sounds like an impressive LCOS company, but I have not seen or heard of their LCOS devices being used in other companies’ products or prototypes.

04:16 Kopin’s Forth Dimensions Display (LCOS)

As discussed earlier with Lighting Silicon, Kopin acquired Ferroelectric LCOS maker Forth Dimension Displays (FDD) in 2011. FDD was originally founded as Micropix in 1988 as part of CRL-Opto, then renamed CRLO in 2004, and finally Forth Dimension Displays in 2005, before Kopin’s 2011 acquisition.

I started working in LCOS in 1998 as the CTO of Silicon Display, a startup developing a VR/AR monocular headset. I designed an XGA (1024 x768) LCOS backplane and the FGA to drive it. We were looking to work with MicroPix/CRL-Opto to do the LCOS assembly (applying the cover glass, glue seal, and liquid crystal). When MicroPix/CRL-Opto couldn’t get their backplane to work, they ended up licensing the XGA LCOS backplane design I did at Silicon Display to be their first device, which they had made for many years.

FDD has focused on higher-end display applications, with its most high-profile design win being the early 4K RED cameras. But (almost) all viewfinders today, including RED, use OLEDs. FDD’s LCOS devices have been used in military and industrial VR applications, but I haven’t seen them used in the broader AR/VR market. According to FDD, one of the biggest markets for their devices today is in “structured light” for 3-D depth sensing. FDD’s devices are also used in industrial and scientific applications such as 3D Super Resolution Microscopy and 3D Optical Metrology.

05:34 Texas Instruments (TI) DLP^®

Around 2015, DLP and LCOS displays seemed to have been used in roughly equal numbers of waveguide-based AR/MR designs. However, since 2016, almost all new waveguide-based designs have used LCOS, most notably the Hololens 1 (2016) and Magic Leap One (2018). Even companies previously using DLP switched to LCOS and, more recently, MicroLEDs with new designs. Among the reasons the companies gave for switching from DLP to LCOS were pixel size and, thus, a smaller device for a given resolution, lower power consumption of the display+asic, more choice in device resolutions and form factors, and cost.

While DLP does not require polarized light, which is a significant efficiency advantage in room/theater projector applications that project hundreds or thousands of lumens, the power of the display device and control logic/ASICs are much more of a factor in near-eye displays that require less than 1 to at most a few lumens since the light is directly aimed into the eye rather than illuminating the whole room. Additionally, many near-eye optical designs employ one or more reflective optics requiring polarized light.

Another issue with DLP is drive algorithm control. Texas Instruments does not give its customers direct access to the DLP’s drive algorithm, which was a major issue for CREAL (to be discussed in the next article), which switched from DLP to LCOS partly because of the need to control its unique light field driving method directly. VividQ (also to be discussed in the next article), which generates a holographic display, started with DLP and now uses LCOS. Lightspace 3D has similarly switched.

Far from giving up, TI is making a concerted effort to improve its position in the AR/VR/MR market with new, smaller, and more efficient DLP/DMD devices and chipsets and reference design optics.

Added 2/21/22: I forgot to discuss the DLP’s new frame rates and field sequential color breakup.

I find the new, much higher frame rates the most interesting. Both DLP and LCOS use field sequential color (FSC), which can be prone to color breakup with eye and/or image movement. One way to reduce the chance of breakup is to increase the frame rate and, thus, the color field sequence rate (there are nominally three color fields, R, G, & B, per frame). With DLP’s new much higher 240Hz & 480Hz frame rates, the DLP would have 720 or 1440 color fields per second. Some older LCOS had as low as 60-frames/180-fields (I think this was used on Hololens 1 – right), and many, if not most, LCOS today use 120-frames/360-fields per second. A few LCOS devices I have seen can go as high as 180-frames/540-fields per second. So, the newer DLP devices would have an advantage in that area.

The content below was extracted from the TI DLP presentation given at AR/VR/MR 2024 on January 29, 2024 (note that only the abstract seems available on the SPIE website).

My Background at Texas Instruments:

I worked at Texas Instruments from 1977 to 1998, becoming the youngest TI Fellow in the company’s history in 1988. However, contrary to what people may think, I never directly worked on the DLP. The closest I came was a short-lived joint development program to develop a DLP-based color copier using the TMS320C80 image processor, for which I was the lead architect.

I worked in the Microprocessor division developing the TMS9918/28/29 (the first “Sprite” video chip), the TMS9995 CPU, the TMS99000 CPU, the TMS34010 (the first programmable graphics processor), the TMS34020 (2nd generation), the TMS302C80 (first image processor with 4 DSP CPUs and a RISC CPU) several generations of Video DRAM (starting with the TMS4161), and the first Synchronous DRAM. I designed silicon to generate or process pixels for about 17 of my 20 years at TI.

After leaving TI, ended up working on LCOS, a rival technology to DLP, from 1998 through 2011. But then when I was designing a aftermarket autmotive HUD at Navdy, I chose use a DLP engine for the projector for its advantages in that application. I like to think of myself as a product focused and want to use whichever technology works best for the given application. I see pros and cons in all the display technologies.

07:25 VueReal MicroLED

VueReal is a Canadian-based startup developing MicroLEDs. Their initial focus was on making single color per device microdisplays (below left).

However, perhaps VueReal’s most interesting development is their cartridge-based method of microprinting MicroLEDs. In this process, they singulate the individual LEDs, test and select them, and then transfer them to a substrate with either passive (wire) or active (ex., thin-film transistors on glass or plastic). They claim to have extremely high yields with this process. With this process, they can make full-color rectangular displays (above right), transparent displays (by spacing the LEDs out on a transparent substrate, and displays of various shapes, such as an automotive instrument panel or a tail light.

I was not allowed to take pictures in the VueReal suite, but Chris Chinnock of Insight Media was allowed to make a video from the suit but had to keep his distance from demos. For more information on VueReal, I would also suggest going to MicroLED-Info, which has a combination of information and videos on VueReal.

08:26 MojoVision MicroLED

MojoVision is pivoting from a “Contact Lens Display Company” to a “MicroLED component company.” Its new CEO is Dr. Nikhil Balram, formerly the head of Google’s Display Group. MojoVision started saying (in private) that it was putting more emphasis on being a MicroLEDs component company around 2021. Still, it didn’t publicly stop developing the contact lens display until January 2023 after spending more than $200M.

To be clear, I always thought the contact lens display concept was fatally flawed due to physics, to the point where I thought it was a scam. Some third-party NDA reasons kept me from talking about MojoVision until 2022. I outlined some fundamental problems and why I thought the contact lens display was a sham in my 2022 Video with Brad Lynch on Mojovision Contact Display in my 2022 CES Discussion video with Brad Lynch (if you take pleasure in my beating up on a dumb concept for about 14 minutes, it might be a fun thing to watch).

So, in my book, Mojovision, the company starts with a major credibility problem. Still, they are now under new leadership and focusing on what they got to work, namely very small MicroLEDs. Their 1.75-micron LEDs are the smallest I have heard about. The “old” Mojovision had developed direct/native green MicroLEDs, but the new MojoVision is developing native blue LEDs and then using quantum dot conversion to get green and red.

I have been hearing about using quantum dots to make full-color MicroLEDs for ~10 years, and many companies have said they are working on it. Playnitride demonstrated quantum dot-converted microdisplays (via Lumus waveguides) and larger direct-view displays at AR/VR/MR 2023 (see MicroLEDs with Waveguides (CES & AR/VR/MR 2023 Pt. 7)).

Mike Wiemer (CTO) gave a presentation on “Comparing Reds: QD vs InGaN vs AlInGaP” (behind the SPIE Paywall). Below are a few slides from that presentation.

Wiemer gave many of the (well-known in the industry) advantages of the blue LED with the quantum dot approach for MicroLEDs over competing approaches to full-color MicroLEDs, including:

• Blue LEDs are the most efficient color
• You only have to make a single type of LED crystal structure in a single layer.
• It is relatively easy to print small quantum dots; it is infeasible to pick and place microdisplay size MicroLEDs
• Quantum dots converted blue to green and red are much more efficient than native green and red LEDs
• Native red LEDs are inefficient in GaN crystalline structures that are moderately compatible with native green and blue LEDs.
• Stacking native LEDs of different colors on different layers is a complex crystalline growth process, and blocking light from lower layers causes efficiency issues.
• Single emitters with multiple-color LEDs (e.g., See my article on Porotech) have efficiency issues, particularly in RED, which are further exacerbated by the need to time sequence the colors. Controlling a large array of single emitters with multiple colors requires a yet-to-be-developed, complex backplane.

Some of the known big issues with quantum dot conversion with MicroLED microdisplays (not a problem for larger direct view displays):

• MicroLEDs can only have a very thin layer of quantum dots. If the layer is too thin, the light/energy is wasted, and the residual blue light must be filtered out to get good greens and reds.
  • MojoVision claims to have developed quantum dots that can convert all the blue light to red or green with thin layers
• There must be some structure/isolation to prevent the blue light from adjacent cells from activating the quantum dots of a given cell, which would cause the desaturation of colors. Eliminating color crosstalk/desaturating is another advantage of having thinner quantum dot layers.
• The lifetime and potential for color shifting with quantum dots, particularly if they are driven hard. Native crystalline LEDs are more durable and can be driven harder/brighter. Thus, quantum dot-converted blue LEDs, while more than 10x brighter than OLEDs, are expected to be less bright than native LEDs
• While MojoVision has a relatively small 1.37-micron LED on a 1.87-micron pitch, that still gives a 3.74-micron pixel pitch (assuming MojoVision keeps using two reds to get enough red brightness). While this is still about half the pixel pitch of the Apple Vision’s Pro ~7.5-micron pitch OLED, a smaller pixel size such as with a single-emitter-with multiple-colors (e.g., Porotech) would be better (more efficient due to étendue see: MicroLEDs with Waveguides (CES & AR/VR/MR 2023 Pt. 7)) for semi-collimating the light using microlenses as needed by waveguides.

10:20 Porotech MicroLED

I covered Porotech’s single emitter, multiple color, MicroLED technology extensively last year in CES 2023 (Part 2) – Porotech – The Most Advanced MicroLED Technology, MicroLEDs with Waveguides (CES & AR/VR/MR 2023 Pt. 7), and my CES 2023 Video with Brad Lynch.

While technically interesting, Porotech’s single-emitter device will likely take considerable time to perfect. The single-emitter approach has the major advantage of supporting a smaller pixel since only one LED per pixel is required. This also results in only two electrical connections (power and ground) to LED per pixel.

However, as the current level controls the color wavelength, this level must be precise. The brightness is then controlled by the duty cycle. An extremely advanced semiconductor backplane will be needed to precisely control the current and duty cycle per pixel, a backplane vastly more complex than LCOS or spatial color MicroLEDs (such as MojoVision and Playnitride) require.

Using current to control the color of LEDs is well-known to experts in LEDs. Multiple LED experts have told me that based on their knowledge, they believe Porotech’s red light output will be small relative to the blue and green. To produce a full-color image, the single emitter will have to sequentially display red, green, and blue, further exacerbating the red’s brightness issues.

12:55 Brilliance Color Laser Combiner

Brilliance has developed a 3-color laser combiner on silicon. Light guides formed in/on the silicon act similarly to fiber optics to combine red, green, and blue laser diodes into a single beam. The obvious application of this technology would be a laser beam scanning (LBS) display.

While I appreciate Brilliance’s technical achievement, I don’t believe that laser beam scanning (LBS) is a competitive display technology for any known application. This blog has written dozens of articles (too many to list here) about the failure of LBS displays.

14:24 TriLite/Trixel (Laser Combiner and LBS Display Glasses)

Last and certainly least, we get to TriLite Laser Beam Scanning (LBS) glasses. LBS displays for near-eye and projector use have a perfect 25+ year record of failure. I have written about many of these failures since this blog started. I see nothing in TriLite that will change this trend. It does not matter if they shoot from the temple onto a hologram directly into the eye like North Focals or use a waveguide like TriLite; the fatal weak link is using an LBS display device.

It has reached the point when I see a device with an LBS display. I’m pretty sure it is either part of a scam and/or the people involved are too incompetent to create a good product (and yes, I include Hololens 2 in this category). Every company with an LBS display (once again, including Hololens 2) lies about the resolution by confabulating “scan lines” with the rows of a pixel-based display. Scan lines are not the same as pixel rows because the LBS scan lines vary in spacing and follow a curved path. Thus, every pixel in the image must be resampled into a distorted and non-uniform scanning process.

Like Brilliance above, TriLites’ core technology combines three lasers for LBS. Unlike Brilliance, TriLites does not end up with the beams being coaxial; rather, they are at slightly different angles. This will cause the various colors to diverge by different amounts in the scanning process. TriLite uses its “Trajectory Control Module” (TCM) to compute how to re-sample the image to align the red, green, and blue.

TriLite then compounds its problems with LBS using a Lissajous scanning process, about the worst possible scanning process for generating an image. I wrote about why the Lissajous scanning process, also used by Oqmented (TriLite uses Infineon’s scanning mirror), in AWE 2021 Part 2: Laser Scanning – Oqmented, Dispelix, and ST Micro. Lissajous scanning may be a good way to scan a laser beam for LiDAR (as I discussed in CES 2023 (4) – VoxelSensors 3D Perception, Fast and Accurate), but it is a horrible way to display an image.

The information and images below have been collected from TriLite’s website.

As far as I have seen, it is a myth that LBS has any advantage in size, cost, and power over LCOS for the same image resolution and FOV. As discussed in part 1, Avegant generated the comparison below, comparing North Focals LBS glasses with a ~12-degree FOV and roughly 320×240 resolution to Avegant’s 720 x 720 30-degree LCOS-based glasses.

Below is a selection (from dozens) of related articles I have written on various LBS display devices:

• 2012 Cynic’s Guild to CES — Measuring Resolution – Discusses how LBS companies confabulate resolution with scan lines
• 2018 North’s Focals Laser Beam Scanning AR Glasses – “Color Intel Vaunt”
• 2015 Celluon Laser Beam Scanning Projector Technical Analysis – Part 1 More on LBS and Resolution:
• 2019 Hololens 2 First Impressions: Good Ergonomics, But The LBS Resolution Math Fails! – This article goes into the basic math behind LBS
• 2020 Hololens 2 Display Evaluation (Part 1: LBS Visual Sausage Being Made) – This article details the Hololens very complex LBS scanning process and its problems
• 2021 AWE 2021 Part 2: Laser Scanning – Oqmented, Dispelix, and ST Micro – Goes into the problems with Lissajous scanning in a display device.
• 2023 Humane AI – Pico Laser Projection – $230M AI Twist on an Old Scam (Title says it all)
• 2016 Wrist Projector Scams – Ritot, Cicret, the new eyeHand
• 2018 CES Haier Laser Projector Watch – (Wrist Projector Scams Revisited)
• 2018 Intel AR “Fixer-Upper” For Sale? Only $350M ???
• 2018 Magic Leap Fiber Scanning Display (FSD) – “The Big Con” at the “Core”

Next Time

I plan to cover non-display devices next in this series on CES and AR/VR/MR 2024. That will leave sections on Holograms and Lightfields, Display Measurement Companies, and finally, Jason and my discussion of the Apple Vision Pro.

Introduction

In part 1, I wrote that I was planning on covering optics and display companies at CES and the SPIE AR/VR/MR conferences in 2024 in part 2 of the video I made with Jason McDowall in this article. However, as I started filling in extra information on the various companies, the article was getting long, so I broke the optics and displays into two separate articles.

In addition to optics companies, I will also be touching on eye track with Tobii, who is doing both optics and eye tracking, and Zinn Labs.

Subscription Options Coming to KGOnTech

Many companies, including other news outlets and individuals, benefit from this blog indirectly through education or directly via the exposure it gives to large and small companies. Many, if not most, MR industry insiders read this blog worldwide based on my conference interactions. I want to keep the main blog free and not filled with advertising while still reporting on large and small companies. To make financial sense of all this and pay some people to help me, I’m in the process of setting up subscription services for companies and planning on (paid) webinars for individuals. If you or your company might be interested, please email subscriptions@kgontech.com.

Outline of the Video and Additional Information

Below is an outline of the second hour of the video, as well as additional comments and links to more information. The times in blue on the left of each subsection below link to the time in the YouTube video discussing a given company.

0:00 Waveguides and Slim Optics

0:03 Schott and Lumus

Schott AG is one of the world’s biggest makers of precision glass. In 2020, Schott entered into a strategic partnership with Lumus, and at AR/VR/MR 2024 and 2023, Lumus was prominently featured in the Schott booth. While Schott also makes the glass for diffractive waveguides, the diffraction gratings are usually left to another company. In the case of the Lumus Reflective waveguides, Schott makes the glass and has developed high-volume waveguide manufacturing processes.

Lumus waveguides consistently have significantly higher optical efficiency (for a given FOV), better color uniformity, better transparency, higher resolution, and less front projection (“eye glow”) than any diffractive waveguide. Originally, Lumus had 1-D pupil-expanding waveguides, whereas diffractive waveguides were 2-D pupil-expanding. The 1-D expanding waveguides required a large projection engine in the non-expanding direction, thus making the projection optics bigger and heavier. In 2021, Lumus first demonstrated their 2-D expanding Maximus prototype waveguides with excellent image quality, 2K by 2K resolution, and 50° FOV. With 2-D expansion, projection image optics could be much smaller. Lumus has continued to advance its Reflective 2D expanding waveguide technology with the “Z-Lens.” Lumus says that variants of this technology could support more than a 70-degree FOV.

Waveguides depend on “total internal reflection” (TIR). For this TIR to work, diffractive waveguides and earlier Lumus waveguides require an “air gap” between the waveguide surface and any other surfaces, including “push-pull” lenses, for moving the waveguide’s apparent focus distance and vision correction. These air gaps can be hard to maintain and source unwanted reflections. Lumus Z-Lens can be embedded in optics with no air gap (and the first waveguide to make this claim) due to the shallower angles of the TIR reflections.

While Lumus waveguides are better than any diffractive waveguide in almost every image quality and performance metric, their big questions have always revolved around volume manufacturing and cost. Schott thinks that the Lumus waveguides can be manufactured in high volume at a reasonable cost.

Over the last ten years, I have seen significant improvements in almost every aspect of diffractive waveguides from many companies (for example, my articles on DigiLens and Dispelix). Diffractive waveguides are easier, less expensive, and much easier to customize. Multiple companies have diffraction waveguide design tools, and there are multiple fabrication companies.

As I point out in the video, many MR applications don’t need the highest image quality or resolution; they need “good enough” for the application. Many MR applications only need simple graphics and small amounts of text. Many applications only require limited colors, such as red=bad, green=good, yellow=caution, and white or cyan for everything else. While others can get away with monochrome (say green-only). For example, many military displays, including night vision, are often monochrome (green or white), and most aviation HUDs are green-only.

I often say there is a difference between being “paid to use” and ” paying for” a headset. By this, I mean that someone is paid to use the headset to help them be more effective in their job, whereas a consumer would be paying for the headset.

For more on Lumus’s 2-D expanding waveguides:

• 2021 Exclusive: Lumus Maximus 2K x 2K Per Eye, >3000 Nits, 50° FOV with Through-the-Optics Pictures

For more on Schott and Lumus’s newer Z-Lens at AR/VR/MR 2023:

• DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8)

For more on green-only (MicroLED headsets) and full-color MicroLEDs through diffractive and Lumus reflective waveguides, see:

• MicroLEDs with Waveguides (CES & AR/VR/MR 2023 Pt. 7)

4:58 Fourier (Metasurface)

Fourier is developing metasurface technology to reflect and redirect light from a projector in the temple area of AR glasses to the eye. If a simple mirror-type coating were placed on the lens, projected light from the temple would bounce off at an angle that would miss the eye.

Multiple companies have previously created holographic Optical Elements (HOEs) for a similar optical function. Luminit developed the HOE used with North Focals, and TruLife Optics has developed similar elements (both Luminit and TruLife’s HOEs are discussed in my AWE 2022 video with Brad Lynch).

Fourier’s metasurface (and HOEs) can act not only as a tilted flat mirror but also as a tilted curved mirror with “optical power” to change magnification and focus. At least in theory (I have not seen it, and Fourier is still in development), the single metasurface would be simpler, compact, and have better optical efficiency than birdbath optics (e.g., Xreal and many others) and lower cost and with much better optical efficiency than waveguides. But while the potential benefits are large, I have yet to see a HOE (or metasurface) with great image quality. Will there, for example, be color uniformity, stray light capture, and front projection (“eye glow”) issues as seen with diffractive waveguides?

Laser beam scanning with direct temple projection, such as North Focals (see below left), uses a Hologram embedded or on the surface of a lens to redirect the light. This has been a common configuration for the lower resolution, small FOV, and very small eyebox Laser Beam Scanning (LBS) glasses shown by many companies, including North, Intel, and Bosch. Alternatively, LCOS, DLP, MicroLED, and laser beam scanning projectors have used waveguides to redirect the light and increase the eyebox size (the eyebox is the range of movement of the eye relative to the glasses where the whole image can be seen).

Avegant (above right), Lumus, Vuzix, Digilens, Oppo, and many others have demonstrated that with waveguides with DLP, LCOS, and MicroLEDs in very small form factors as HOEs and Metasufaces (see DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8). Still, waveguides are much lower in efficiency, so much so that the use of MicroOLED is impractical with waveguides. In contrast, using MicroOLED displays is possible with HOEs and Fourier’s metalenses. There are also potential differences in how prescription lenses could be supported.

As discussed above, holographic mirrors can also be used to form the equivalent of a curved mirror that is also tilted. The large CREAL3D prototype (below left) shows the two spherical semi-mirrors. CREAL3D planes to replace these physical mirrors with a flat HOE (below right).

Fourier metalens would perform the same optical function as the HOE. We will have to wait and see the image quality and whether there are significant drawbacks with either HOEs or metalenses. My expectation is that both metalenses and HOEs will have similar issues as diffraction gratings.

Some related articles and videos on small form factor optics and Videos.

• North’s Focals Laser Beam Scanning AR Glasses – “Color Intel Vaunt”
• Intel AR Glasses Follow-Up
• AWE 2021 Part 2: Laser Scanning – Oqmented, Dispelix, and ST Micro
• DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8
• Intel AR “Fixer-Upper” For Sale? Only $350M ???
• 06:49 Innovative LCOS Optics (Digilens, Avegant, Lumus, Magic Leap)
• 14:40 Avegant (Small LCOS Engine)

6:23 Morphonics

Morphontonics has developed methods for making waveguides and similar diffractive structures on large sheets of glass. They can make many small diffractive waveguides at a time or fewer large optical devices. In addition to waveguides, Morphotonics makes a light guide structure for the Leia Lightfield monitor and tablet.

Morphotonics presentation at AR/VR/MR 2023 can be found here: Video of Morphotonics AR/VR/MR 2023 presentation.

10:33 Cellid (Wave Guides)

Cellid is a relatively new entrant in waveguide making. I have seen their devices for several years. As discussed in the video, Cellid has been continually improving its waveguides. However, at least at present, it still seems to be behind the leading diffractive waveguide companies in terms of color uniformity, FOV, and front projection (“eye glow).

11:47 LetinAR

Several companies are using LetinAR’s PinTilt optics in the AR glasses. At CES, JorJin was showing their J8L prototypes in the LetinAR booth. Nimo (as discussed in Mixed Reality at CES and the AR/VR/MR 2024 Video (Part 1 – Headset Companies) was showing their LentinAR-based glasses in their own booth. Sharp featured their LentinAR glasses in their booth but didn’t mention they were based on LetinAR optics.

LetinAR’s optics were also used in an AT&T football helmet display application for the deaf (upper left below).

LetinAR originally developed “pin mirror” optics, which I first covered in 2018 (see CES 2018 in the listings below). The pin-mirror technology has evolved into their current “PinTilt” technology.

While LetinAR has several variations of the PinTilt, the “B-Type” (right) is the one I see being used. They use an OLED microdisplay as the display device. The image light from the OLED makes a TIR (total internal reflection) bounce off the outside surface into a collimating/focusing mirror and then back up through a series of pupil-replicating slats. The pupil replication slats enable the eye to move around and support a larger FOV.

As I discussed in the video, the image quality is much better than with the Pin-Mirrors, but gaps can be seen if your eye is not perfectly placed relative to the slats. Additionally, with the display off, the view can be slightly distorted, which can likely be improved in the manufacturing process. LetinAR also let me know that they are working on other improvements.

LetinAR’s PinTilt is much more optically efficient than diffractive or even Lumus-type reflective waveguides, as evidenced by its use of micro-OLEDs rather than much brighter LCOS, DLP, or micro-LEDs. At the same time, they offer a form factor that is close to waveguides.

Some other articles and videos covering LetinAR:

• CES 2023 Video about LetinAR pin mirrors with Brad Lynch (with picture from CES 2020)
• CES 2019 – AR and other Interesting Display Technology
• CES 2018 (Part 1 – AR Overview)

13:57 Tooz

Tooz was originally spun out of Zeiss Group in 2018, but in March 2023, they returned to become part of Zeiss. Zeiss is an optical giant founded in 1846 but is probably most famous to Americans as the company making the inserts for the Apple Vision Pro.

Tooz’s “Curved Waveguide” works differently than diffractive and Lumus-type reflective waveguides, which require the image to be collimated, use many more TIR light bounces, and have pupil replication. Strictly speaking, none of these are”waveguides,” but the diffractive and Lumus-type devices are what most people in the industry call waveguides.

The Tooz device molds optics and a focusing mirror to move the focus of the display device, which currently can be either a Micro-OLED or, more recently, (green only) Micro-LED. The image light then makes a few TIR bounces before hitting a Fresnel semi-mirror, which directs the light toward the user’s eye (above right). The location of the Fresnel semi-mirror, and thus the image, is not centered in the user’s field of view but slightly off to one side. It is made for a monocular (single-eye) display. The FOV is relatively small with 11- and 15-degree designs.

Tooz’s Curved Waveguide is aimed at data snacking. It has a small FOV and a Monocular display off the side. The company emphasizes the integration of prescription optics and the small and lightweight design, which is optically much more efficient than other waveguides.

Tooz jointly announced just before the AR/VR/MR conference that they were working with North Ocean Photonics to develop push-pull optics to go with waveguides. Tooz, in their AR/VR/MR 2024 presentation, discussed how they were trying to be the prescription optics provider for both their curved waveguides and what they call planar waveguides. One of their slides demonstrated the thickness issue with putting a push/pull set of lenses around a flat waveguide. The lenses need to be thicker to “inscribe” the waveguide due to their curvature (below right).

19:08 Oorym

Oorym is a small startup founded by Yaakov Amitai, a founder and former CTO of Lumus. Oorym has a “waveguide” with many more TIR bounces than Tooz’s design but many less than diffractive and Lumus waveguides. They use a Fresnel light redirecting element. It does not require collimated light and is much more efficient than other waveguides. They can support more than a 50-degree FOV. It is thicker and more diffractive, and Lumus waveguides are in the same order as the thickness of LetinAR. Oorym is also developing a non-head-mounted Heads-Up Display (HUD) device.

Oorym

21:57 Gixel

Gixel’s technology has to be among the most “different” I have seen in a long time. The concept is to have a MicroLED “bar” display with only a single or a few rows of pixels in one direction and with the full horizontal resolution in the other. The “rows” may have full-color pixels or a series of 3 single-color row arrays. Then, a series of pupil-replicating slats rotate to scan the bar/row image vertically synchronously with a time-sequential change of the row display. In this way, the slats scan row display forms a whole image to the eye (and combines the colors if there are separate displays for each color).

They didn’t have a full working prototype, but they did have the rotating slats working.

My first impression is that it has a Steampunk feel to the design. I can see a lot of issues with the rotating slats, their speed and vibration, the time-sequential display, and a mirage of other potential issues. But still, it wins for the sheer audacity of the approach.

23:42 Meta Research (Time Sequential Fixed Foveated Display) & Varjo

Meta Research presented the concept of a time-sequence fixed-foveated display using single pancake optics. The basic idea is that pancake optics work by making two passes through some of the refractive and mirror optics, which magnifies the display. In a normal pancake, quarter waveplates change the light’s polarization and affect the two passes. A (pixel-less) liquid crystal shutter can act as a switchable quarter waveplate. This way, the display light will make one or two passes through part of the optics to cause two different magnifications. By time sequencing the display with the LC shutter’s switching, both a lower angular resolution but larger image and a higher angular resolution but smaller “foveated” display will be seen by the eye time sequentially.

This basically happens with a single set of optics and a single display, which is what Varjo was doing with their “fixed foveated display,” which used two display devices, optics, and a combining beam splitter.

I like to warn people that when a research group from a big company presents a concept like this to all their competitors at a conference like AR/VR/MR, it is definitely NOT what they are doing in a product.

Fixed (and Eye Tracking) Foveated Displays

In 2017, Varjo was focused on its foveated display technology. Their first prototype had a “fixed foveated display,” meaning the central high-resolution region didn’t move. Varjo claimed they would soon have the foveated display tracking the eye, but as far as I know, they never solved the problem.

It turns out that tracking the eye and moving the display is a seemingly impossible problem to solve with the eye’s saccadic movement, even with exceptional eye tracking. As I like to say, “While eye tracking may know where the eye is pointing, you don’t know what the eye has seen.” Originally, researchers thought that human vision fully blanks with saccadic movement, but later research suggests that it only semi-blanks out vision with movement. Combined with the fact that what a human “sees” is basically a composite of multiple eye positions, making a foveated display that tracks the eye is exceedingly difficult, if not impossible. The problem with artifacts due to eye movement, such as field sequential color breakup, they will tend to appear as flashes that are distracting.

We are seven years since Varjo told me they were close to solving the eye-tracking foveated display. Varjo figured out that about 90% of the benefit of a moving foveated display could be realized with a fixed foveated display near the center of the FOV. They may also have realized that solving the problems with a moving foveated display was more difficult than they thought. Regardless, Varjo has pivoted from being a “foveated display company” to a “high-resolution VR/MR company” aimed primarily at enterprise applications. Pixel sizes and resolution of display devices improved to the point where it is now better to use a higher resolution display than to combine two displays optically.

Eyeway Vision Foveated Display (and Meta)

In 2021, I visited Eyeway Vision, which also worked on foveated displays using dual laser scanning displays per eye. After an acquisition by Meta fell through, Eyeway Vision went bankrupt. Eyeway Vision had a fixed foveated display and sophisticated eye tracking, but it went bankrupt before solving the moving foveated display.

Eyeway Vision’s founder, Boris Greenburg, has recently joined VoxelSensors, and VoxelSensors is looking at using their technology for eye/gaze tracking and SLAM (see Zinn Labs later)

Foveated Display (ex., Varjo) vs. Foveated Rending (ex., Apple Vision Pro)

I want to be clear between foveated rendering, where the display is fixed, and just the level of detail in the rendering changes based on eye tracking, from a foveated display, where a high-resolution sub-display is inset within a lower resolution display. Foveated rendering such as the Apple Vision Pro or Meta Quest Pro is possible, although today’s implementations have problems. However, it may be impossible to have a successful eye-tracking foveated display.

For more on this blog’s coverage of Foveated Displays, see:

• 2017 Varjo Foveated Display (Part 1)
• 2017 Varjo Foveated Display Part 2 – Region Sizes
• 2018 Varjo Foveated Display and Passthrough AR at AWE 2018
• 2021 Exclusive: EyeWay Vision Part 1: Foveated Laser Scanning Display
• 2021 Exclusive: EyeWay Vision Part 2: Through the Optics Pictures
• 2021 EyeWay Vision Part 3: Analysis

32:05 Magic Leap (Mostly Human Factors)

At AR/VR/MR 2024, Magic Leap gave a presentation that mostly discussed human factors. They discussed some issues they encountered when developing the Magic Leap One, including fitting a headset to a range of human faces (below right). I thought the presentation should have been titled “Why the Apple Vision Pro is having so many problems with fitting.”

In 2016, This Blog Caught Magic Leap’s Misleading Video

In showing Magic Leap’s history, they showed a prototype headset that used birdbath optics (above left). Back in 2016, Magic Leap released a video that stated, “Shot directly through Magic Leap technology . . . without the use of special effects or compositing.” I noted at the time that this left a lot of legal wiggle room and that it might not be the same “technology” they would use in the final product, and this turned out to be the case. I surmised that the video used OLED technology. It’s also clear from the video that it was not shot through a waveguide. It appears likely that the video was shot using an OLED through birdbath optics, not with the Waveguide Optics and LCOS display that the Magic Leap One eventually used.

In 2019, Magic Leap sued (and lost to) Nreal (now Xreal), which developed an AR headset using birdbath optics and an OLED display. Below are links to the 2016 article analyzing the Magic Leap deceptive video and my 2020 follow-up article:

• 2016 Magic Leap – The Display Technology Used in their Videos
• 2020 Magic Leap Starts to Auger-In with ~3.7X the Money as Theranos – I Tried to Warn Everyone

36:45 NewSight Reality (Not Really “Transparent” MicroLED)

Sorry for being so blunt, but NewSight Reality’s “transparent” MicroLED concept does not and will not ever work. The basic concept is to put optics over small arrays of LEDs, and similar to pupil replication, the person will see an image. It is the same “physics” as MojoVision’s contact display (which I consider a scam). In fact, NewSight’s prototype has nine MojoVision displays on a substrate (below center)

The fundamental problem is that to get a display of any resolution, plus the optics, the “little dots” are so big that they, combined with diffraction, cause a blurry set of gray dots in a person’s vision. Additionally, the pupil replication effect ends up with a series of circles where you can see the image.

38:55 Other Optics and Eye Tracking

The next section is on other optics and eye tracking. Thanks to Tobii being involved in both, they sort of tie this section together.

39:01 AddOptics

AddOptics developed a 3-D-printed optical mold process. It was founded by former Luxexcel employees (Luxexcel was subsequently acquired by Meta in 2022).

I covered AddOptics last year in CES 2023 (Part 3)—AddOptics Custom Optics. The big addition in 2024 was that they showed their ability to make push-pull optics for sandwiching a waveguide. They showed they could support waveguides that required an air gap or not. As far as I am aware, most, if not all, diffractive waveguides require an air gap. The only waveguide I know of that claims they don’t need an air gap is the newer Lumus reflective-based waveguide (discussed in a previous article). Still, I have not heard of whether AddOptics is working with Lumus or one of Lumus’s customers.

Luxexcel had developed a process to directly 3-D print optics without the need for any resurfacing. This means they need to print very fine layers very precisely, lens by lens. While it means each lens it custom can be custom fit, it also seems to be an expensive process compared to the way prescription lenses are made today. By making “low run” 3-D printed molds (something that Luxexcel could also do), AddOptics would have a lower cost per unit and a faster approach. It would require having a stock of molds, but it would not require a prohibitive number of molds to support most combinations of diopter and cylinder (astigmatism) correction.

42:12 Tobii

Tobii, founded in 2001, has long been known for its eye-tracking technology. Tobii was looking to embed LED illuminators in lenses and was working with Interglass. When Interglass (founded in 2004) went bankrupt in 2020, Tobii hired the key technical team members from Integlass. Meta Materials (not to be confused with Meta, formerly Facebook) acquired the assets of Interglass and is also making a similar technology.

The Interglass/Tobii/Meta-Materials process uses many glass molds to support variations of diopter and cylinder adjustments for prescriptions. The glass molds are injected with UV-cured plastic resin, which, after curing, forms lens blanks/rounds. When molding, the molds can be rotated to set the cylinder angle. The round lens blanks can then be cut by conventional lens fitting equipment.

At 2023’s AR/VR/MR, Tobii demonstrated (left two pictures below) how their lenses were non-birefringent, which is important when working with polarized light-based optics (e.g., Pancake Optics, which Tobii says they can make) and displays (LCDs and LCOS). Tobii has videos on its website that show the lens-making and electronic integrating process (below right).

43:44 Zinn (and VoxelSensors)

Zinn Labs uses a Prophesee event-based camera sensor (Zinn and Prophesee announcement). The Prophesee event camera sensor was jointly developed with Sony. Zinn uses Prophesee’s 320×320 6.3μm pixel BSI (BackSide Illuminated) event-based sensor in a 1/5” optical format.

Event camera pixels work like the human eye in detecting changes rather than the absolute value of each pixel. The pixels are much more complex than a conventional camera sensor, with photodiodes and comparators integrated into each pixel using Sony’s BSI process. Rather than scanning out the pixel value at a frame rate, each pixel reports when it changes significantly (more details can be found in the Prophesee white paper – free, but you have to give an email address). The advantage of the event camera in image recognition is that it tends to filter out/ignore everything that is not changing.

Zinn Labs has developed algorithms that then take the output from the event camera and turn it into where the eye is gazing (for more information, see here).

VoxelSensors (and Zinn Labs)

VoxelSensors has a very different type of event sensor called a “SPAES (Single Photon Active Event Sensor)” that could be used for eye/gaze tracking. Quoting from VoxelSensors:

VoxelSensors leverages its distinctive SPAES (Single Photon Active Event Sensor) technology, allowing the integration of multimodal perception sensors, such as innovative hand and gaze tracking and SLAM, with high precision, low power consumption, and low latency. Fusing these key modalities will enable the development of next-gen XR systems.

As discussed earlier, VoxelSensors also recently hired Eyeway Vision found Boris Greenberg, who has extensive experience in eye/gaze tracking.

VoxelSensors’s SPAES uses a laser scanner to scan the area of interest in a narrow-band infrared laser (where the Prophesee event camera would use IR LED flood illumination) and then detect the laser scanner’s return to the area of interest. With narrow-band filtering to filter out all but the laser’s wavelength, the SPAES is designed to be extremely sensitive (they claim as little as a single photon) to the laser’s return. Like the Prophesee event camera, the VoxelSensors’s SPAES returns the pixel location when an event occurs.

While the VoxelSensor’s pixel is more complex than a traditional sensor, it seems simpler than Prophesee’s event camera pixel, but then VoxelSensor requires scanning lasers versus LED. Both are using event sensors to reduce the computational load. I have no idea at this point which will be better at eye tracking.

VoxelSensors with one or more sets of laser scanners and sensors can detect in three dimensions, which is obviously useful for SLAM but might also have advantages for eye tracking.

For more on Voxel Sensors my 2023 CES article: CES 2023 (4) – VoxelSensors 3D Perception, Fast and Accurate.

44:13 Lumotive (LCOS-Based Laser Scanning for LiDAR)

Lumotive has a technology that uses LCOS devices to scan a laser beam. Today, LiDAR systems use a motor-driven rotating prism or a MEMs mirror to scan a laser beam, resulting in a fixed scanning process. The Lumotive method will let them dynamically adjust and change the scanning pattern.

46:03 GreenLight Optics

I’ve known Green Light Optics since its founding in 2009 and have worked with them to help me with several optical designs over the years. Greenlight can design and manufacture optics and is located in Cincinnati, Ohio. I ran into GreenLight at the Photonics West exhibit following the AR/VR/MR conference. I thought it would be helpful for other companies that might need optical design and manufacturing to mention them.

Quoting GreenLights website:

“Greenlight Optics is an optical systems engineering and manufacturing company specializing in projection displays, LED and laser illumination, imaging systems, plastic optics, and the integration of optics with electrical and mechanical systems.”

Next Time – Display Devices and Test and Measurement Companies

In the next part of this series will on CES and AR/VR/MR 2023, I plan to cover display devices and a few test and measurement companies.

Introduction

Today, rather than shooting through AR/MR headsets, I pointed my camera skyward toward the Solar Eclipse. They were forecasting a lot of cloud cover in the Dallas area, and it looked bad less than two hours before the eclipse was to start. But then, as the eclipse neared, the clouds mostly parted so I could get some good pictures between the times when clouds moved by.

But the sun was still behind a cloud as it was about time for the total eclipse. Fortunately, a few moments into the totality, the clouds moved out of the way.

All the images were cropped without any scaling. To see them in full size, you can click on any image.

During the eclipse, I took pictures between the times when the clouds were blocking the view. At one point, not long after totality, there was a fairly dark low cloud and some light wispy clouds. They blocked enough sunlight that I could take pictures without the sun filter, yet I could still see the eclipse. The clouds had an interesting effect. It looked like a shot of a crescent moon through the clouds.

Photography Info

For the camera people in the audience, the pictures were taken with a Canon R5 (45MP) camera with an RF 100-500mm lens at 500mm with a 1.4x teleconverter (netting 700mm). That is about half what it would take to get the full sun and corona to fill the frame, so all the pictures below are cropped (except for the last picture with the cloud cover). Most of the pictures were shot on a tripod with a “geared head” (the same one I use to line up the camera to shoot through AR and MR headset), so once I had the sun lined up in the camera’s LCD display, I only had to turn a couple of knobs occasionally. Except during totality and a few shots when the clouds were blocking, I used a sun filter (Silver-black polymer). All shots were taken with exposure bracketing (3 shots at different exposures).

Update 4/2/2024: Everysight corrected a comment I made about the size of their eyebox.

Introduction

This blog has covered mixed reality (MR) headsets, displays, and optics at CES since 2017 and SPIE’s AR/VR/MR conference since 2019. Both conferences occur in January each year. With this blog’s worldwide reputation (about half of the readers are from outside the U.S.), many companies want to meet. This year, I met with over 50 companies in just one month. Then Apple released the Apple Vision Pro on Feb. 2nd.

As this blog is a one-person operation, I can’t possibly write in detail about all the companies I have met with, yet I want to let people know about them. Last year, in addition to articles on some companies, Brad Lynch of the SadlyIsBradley YouTube channel and I made videos about many companies I met at CES 2023. Then, for AR/VR/MR 2023, I wrote an eight (8) part series of articles on AR/VR/MR. For CES 2024, I wrote a three (3) part series covering many companies.

However, with my Apple Vision Pro (AVP) coverage plus other commitments, I couldn’t see how to cover the over 50 companies I met with in January. While the AVP is such a major product in mixed reality and is important for a broad audience, I don’t want the other companies working on MR headsets, displays, and optics to be forgotten. So, I asked Jason McDowall of The AR Show to moderate a video presentation of the over 50 companies, with each company getting one slide.

Jason and I recorded for about 4 hours (before editing), split over two days, which works out to less than 5 minutes per company. This first hour of the video covers primarily headset companies. I made an exception for the combination of Avegant’s prototype that used Dispelix as it seemed to fit with the headsets.

In editing the video, I realized my presentation was a little “thin” regarding details on some companies. I’m adding some supplementary information and links to this article. I also moved a few companies around in the editing process and re-recorded a couple of sections, so the side numbers don’t always go in order.

Subscription Options Coming to KGOnTech

Between travel expenses and buying an Apple Vision Pro (AVP) with a MacBook for testing the AVP, I spent about $12,000 out of pocket in January and early February alone. Nobody has ever paid to be included (or excluded) in this blog. This blog, which started as a part-time hobby, has become expensive in terms of money and a full-time job. What makes it onto the blog is the tip of the iceberg of time spent on interviews, research, photographing and editing pictures and videos, and travel.

Many companies, including other news outlets and individuals, benefit from this blog indirectly through education or directly via the exposure it gives to large and small companies. Many, if not most, MR industry insiders read this blog worldwide based on my conference interactions. I want to keep the main blog free and not filled with advertising while still reporting on large and small companies. To make financial sense of all this and pay some people to help me, I’m in the process of setting up subscription services for companies and planning on (paid) webinars for individuals. If you or your company might be interested, please email subscriptions@kgontech.com.

Outline of the Video and Additional Information

Below is an outline of the first hour of the video, along with some additional comments and links to more information. The times in blue on the left of each subsection below are the times in the YouTube video discussing a given company.

0:00 Jason McDowall of the AR Show and Karl Guttag of KGOnTech introductions.

Jason and I briefly introduced ourselves.

2:59 Mixed Reality Major Design Challenges

My AR/MR design challenge list started with 11 items in a guest article in Display Daily in December 2015 with Sorry, but there is no Santa Claus – Display Daily. Since then, the list has grown to 23.

The key point is that improving any of these items will negatively affect multiple other items. For example, having a wider field of view (FOV) will make the optics bigger, heavier, and more expensive. It will also require a higher resolution display to support the same or better angular resolution, which, in turn, means more pixels requiring more processing, which will need more power, which means bigger batteries and more thermal management. All these factors combine to hurt cost and weight.

6:34 Xreal (Formerly Nreal)

I’ve followed Nreal (now Xreal) since its first big splash in the U.S. at CES 2019 (wow, five years ago). Xreal claims to have shipped 300,000 units last year, making it by far the largest unit volume shipper of optical AR headsets.

At CES 2024, Xreal demonstrated a future design that goes beyond their current headsets and adds cameras for image recognition and SLAM-type features.

BMW invited me to a demo of their proof-of-concept glasses-based heads-up display. The demo used Xreal glasses as the display device. BWM had added a head-tracking device under its rearview mirror to lock the user’s view of the car.

But even at CES 2019, Nreal was a case of déjà vu, as it looked so much like a cost-reduced version of the Osterhaut Design Group (ODG) R-9 that I first saw at CES 2017 and started covering and discussing in 2016. The ODG R-9 and the original X-Real had similar birdbath designs and used a Sony 1920×1080 Micro-OLED display. According to a friend of this blog and a former ODG R-9 designer and now CEO of the design firm PulsAR, David Bonelli, there are still some optical advantages of the ODG R-9 that others have yet to copy.

Below is a link to my recent article on CES, which discusses Xreal and my ride wearing the BMW AR demo. I have also included some links to my 2021 teardown of the Nreal birdbath optics and 2016 and 2017 articles about the ODG-R9.

• (2024) CES (Pt. 3), Xreal, BMW, Octrux, Nimo Planet, Sightful, and LetinAR
• (2021) Nreal Teardown: Part 1, Clones and Birdbath Basics
• (2021) Nreal Teardown: Part 2, Detailed Look Inside
• (2021) Nreal Teardown: Part 3, Pictures Through the Lens
• CES 2019 – AR and other Interesting Display Technology (including Nreal)
• (2016) AR/MR Optics for Combining Light for a See-Through Display (Part 1) Early ODG R-9 prototype called “Horizons.”
• (2017) Near-Eye Bird Bath Optics Pros and Cons – And IMMY’s Different Approach Includes discussion of ODG’s R-9 and Birdbath Optics

11:48 Vuzix

Vuziz was founded in 1997 before making see-through AR devices, no less waveguides, became practical. It now has a wide range of products aimed at different applications. Vuzix founder and CEO Paul Travers has emphasized the need for rugged, all-day wearable AR glasses.

Vuzix historically has primarily had small, lightweight designs, with most later products having a glasses-like form factor. Vuzix originally licensed waveguide technology from Nokia, the same technology Microsoft licensed and later acquired for its Hololens 1. Vuzix says its current waveguide designs are very different from what it licensed from Nokia.

Vuzix’s current waveguide-based products include the monocular BLADE and the biocular SHIELD, which use Texas Instruments DLP displays.  Vuzix ‘s latest products are the Ultralight and Ultralight-S, which use Jade Bird Display MicroLEDs driving a waveguide. The current monocular designs use a green-only Jade Bird Display (JBD) with a 640 by 480 resolution and weigh only 34 grams. Vuzix has also announced plans to partner with the French startup Atomistic to develop full-color on a single device, MicroLEDs.

Multiple companies use Vuzix glasses as the headset platform to add other hardware and software layers to make application AR headsets. Xander was at CES with their AI voice-to-text glasses (discussed later). The company 3D2Cut has AI software that shows unskilled workers where to prune wine grape vines based on inputs from vine pruning experts. At last year’s CES, I met with 360world and their ThermalGlass prototype, which added thermal cameras to a Vuzix headset.

Below are links to my 2024 CES article that included Vuzix, plus a collection of other articles about Vuzix from prior years:

• (2024) CES (Pt. 2), Sony XR, DigiLens, Vuzix, Solos, EverySight, Mojie, TCL color µLED
• DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8)
• CES 2023 SadlyItsBradley Videos Part 1-4 and Meta Leak Controversy
• AWE 2022 (Part 6) – MicroLED Microdisplays for Augmented Reality
• (2021) News: WaveOptics & Jade Bird Display MicroLED Partnership
• CES 2017 AR, What Problem Are They Trying To Solve?
• AR Longan Vision AR for First Responders (CES – AR/VR/MR 2023 Pt. 5) (ThermalGlass added an infrared camera to Vuzix Blade).

17:13 Digilens

I’ve met with Digilens many times through the years. This year was primarily an update and improvements on this major announcement of their Argo headset from last year (see 2023 article and video via the links below).

Digilens said that in response to my comments last year, they designed an Argo headband variant with a rigid headband that does not rest on the nose and can be flipped up out of view. This new design supports wearing ordinary glasses and is more comfortable for long-term wear. Digilens said many of their customers like this new design variation. A major problem I see with the Apple Vision Pro is the way it is uncomfortably clamped to the face and that it does not flip up like, say, the Lynx MR headset (see also video with Brad Lynch) and Sony MR Headset announced at CES 2024 (which looks very much like the Lynx headset).

Digilens also showed examples of their one-, two-, and three-layer waveguides, which can trade in weight and cost for differences in image quality. They also showed examples of moving the exit grating to different locations in the waveguide.

As I have covered Digilens so much in the past (see links below for some more recent articles), this year’s video was just an update:

• (2024) CES (Pt. 2), Sony XR, DigiLens, Vuzix, Solos, EverySight, Mojie, TCL color µLED
• DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8)
• CES 2023 SadlyItsBradley Videos Part 1-4 and Meta Leak Controversy
• AWE 2022 (Part 6) – MicroLED Microdisplays for Augmented Reality
• AWE 2022 Review/Discussion Video Part 5
• DigiLens Visit

20:00 Avegant

Avegant has become a technology development company. They are currently focused on designing small LCOS engines for AR glasses. They presented an update at the AR/VR/MR 2024 conference. Right before the conference, Avegant announced its development of “Spotlight™” to improve contrast by selective illumination of the LCOS panel, similar to LED array LCD TVs with local dimming.

Avengant has shown a very small 30-degree FOV, LCOS-based, 1280×720 pixel, light engine supporting a glasses-like form factor. Avegant’s glasses designs support higher resolution, larger FOV, and a smaller form factor than laser beam scanning or X-Cube-based MicroLEDs (see TCL below). They also got over 1 million nits out of their 30-degree FOV engines. While Avegant designed and built the projector engine and prototype glasses, they used Dispelix waveguides (to be discussed next).

Below are links to blog articles about Avegant’s small LCOS engines:

• DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8)
• AWE 2022 (Part 6) – MicroLED Microdisplays for Augmented Reality
• AWE 2022 Video Review – KGOnTech YouTube (with SadlyItsBradley)
• AWE 2022 (Part 7) Video – Magic Leap 2 with some Hololens 2
• Magic Leap 2 at SPIE AR/VR/MR 2022

24:46 Dispelix (and Avegant)

Dispelix is a waveguide design company, not a headset maker. Avegant, among others, was using Dispelix waveguides (and why they were discussed at this point in the video).

Dispelix presented at the AR/VR/MR conference, where they discussed their roadmap to improve efficiency, reduce “eye glow,” and reduce “rainbow artifacts” caused by diffraction grating light capture.

Dispelix claims to have a roadmap to improve light throughput by a factor of ~4.5 over its current Selva design.

Dispelix, like several other diffractive waveguide companies, including Vuzix and Digilens, uses pantoscopic (front to back) tilt to reduce the eye glow effect, which is common with most other diffractive waveguides (most famously, Hololens). It turns out that for every one-degree of tilt, the “glow” is tilted down by two degrees such that with just a few degrees of tilt, the glow is projected well below most people’s view. Displelix has said that a combination of grating designs and optical coatings can nearly eliminate the glow in future designs.

Another problem (not discussed in the video) that has plagued diffractive waveguides has been the “rainbow artifact” caused by external light, particularly overhead from in front or behind the waveguide, being directed to the eye from the diffraction gratings. Because the gratings effect is wavelength-dependent, the light is broken into multiple colors (like a rainbow). Dispelix says they are developing designs that will direct the unwanted external light away from the eye.

(2024) CES (Pt. 2), Sony XR, DigiLens, Vuzix, Solos, EverySight, Mojie, TCL color µLED

• DigiLens, Lumus, Vuzix, Oppo, & Avegant Optical AR (CES & AR/VR/MR 2023 Pt. 8)
• MicroLEDs with Waveguides (CES & AR/VR/MR 2023 Pt. 7)
•  AWE 2021 Part 2: Laser Scanning – Oqmented, Dispelix, and ST Micro

30:50 Tilt-Five (and CEO Jeri Ellsworth)

I met with Jeri Ellsworth, the CEO of Tilt-Five, at CES. In addition to getting an update on Tilt-Five (with nothing I can’t talk about), Jeri and I discussed our various histories working on video game hardware, graphics co-processors, and augmented reality.

BTW, Jeri Ellsworth, Jason McDowall, Adi Robertson (editor at The Verge), Ed Tang (CEO of Avegant), and I are slated to be on a panel discussion at AWE 2024.

Below are some links to my prior reporting on Tilt-Five.

• AWE 2021 Part 1 – Tilt-5 Was Magical
• CES 2023 SadlyItsBradley Videos Part 1-4 and Meta Leak Controversy (1:25:43 Tilt Five)

36:05 Sightful Spacetop

Sightful’s Spacetop is essentially a laptop-like keyboard and computer with Xreal-type birdbath optics using 1920×1080 OLED microdisplays with a 52-degree FOV. Under the keyboard are the processing system (Qualcomm Snapdragon XR2 Kryo 585TM 8-core 64-bit CPU and AdrenoTM 650 GP), memory (8GB), flash (128GB), and battery (5 hours of typical use). The system runs a “highly modified” Android operating system.

I saw Sightful at the Show Stoppers media event at CES, and they were nice enough to bring me custom prescription inserts to the AR/VR/MR conference. Sightful’s software environment supports multiple virtual- monitors/windows of various sizes, which are clipped to the glasses’ 1920×1080, 52-degree view. I believe the system uses the inertial sensors in the headset to make the virtual monitors appear stationary as opposed to the more advanced SLAM (simultaneous localization and mapping) used by many larger headsets.

As a side note, my first near-eye-display work in 1998 was on a monocular headset to be used with laptops as a private display when traveling. I designed the 1024×768 (high resolution for a 1998 microdisplay) LCOS display device and its controller. The monocular headset used color sequential LED illumination with birdbath mirror optics. Given the efficiency and brightness of LEDs of the day, it was all we could do to make a non-see-through monocular device. Unfortunately, the dot-com bust happened in 1999, which took out many high-tech startups.

I wrote about Sightful in my 2024 CES coverage:

• (2024) CES (Pt. 3), Xreal, BMW, Octrux, Nimo Planet, Sightful, and LetinAR

36:05 Nimo

Nimo’s “Spatial Computing” approach is slightly different from Sightful’s. Instead of combining the computing hardware with the keyboard like Sightful, Nimo has a small computing and battery module that works as a 3-D spatial mouse with a trackpad (on top). Nimo has a USB-C connection for AR glasses, WiFi 6, and Bluetooth 5.1 for communication with an (optional) wireless keyboard.

The computing specs resemble Sightful’s, with a Qualcomm® XR2 8-core CPU, 8GB RAM, and 128GB Storage. Nimo supports working with Rokid, Xreal, and its own LetinAR-Optics-based 1920×1080 OLED AR glasses via its USB-C port, which provides display information and power.

Like Sightful, Nimo has a modified Android Operating system that supports multiple virtual monitors/windows. It uses the various glasses’ internal sensors to detect head movement to keep the monitors stationary in 3-D space as the user’s head moves.

I wrote about Nimo Planet in my 2024 CES coverage:

• (2024) CES (Pt. 3), Xreal, BMW, Octrux, Nimo Planet, Sightful, and LetinAR

38:59 .Lumen (headset for the blind)

Lumen is a headset for blind people that incorporates lidar, cameras, and other sensors. Rather than outputting a display image, it provides haptic and audible feedback to the user. I don’t know how to judge this technology, but it seems like an interesting case where today’s technology could help people.

40:07 Ocutrx Oculenz

Ocutrx’s OcuLenz was initially aimed at helping people with macular degeneration and other forms of low vision. However, at the Ocutrx booth on its website at the CES ShowStoppers event, Ocutrx emphasized that the headset could be used for more than low vision, including gamers, surgeons, and military personnel. The optical design was done by an old friend, David Kessler, whom I ran into at the Ocutrx booth at CES and the AR/VR/MR conference.

The Oculenz uses larger-than-typical birdbath optics to support a 72-degree (diagonal) FOV. It uses 2560 x 1440 pixels per eye, so they will have a similar angular resolution but wider FOV than the more common 1920×1080 birdbath glasses (e.g., Xreal), which typically have 45- to ~50-degree FOVs. Unlike the typical birdbath glasses, which have separate processing, the Oculenz integrates a Qualcomm Snapdragon® XR2 processor, Wi-Fi, and cellular connectivity. This headset was originally aimed at people with low vision as a stand-alone device.

I wrote about Ocutrx and some of the issues of funding low-vision glasses in my earlier report on CES 2024, linked below:

• (2024) CES (Pt. 3), Xreal, BMW, Ocutrx, Nimo Planet, Sightful, and LetinAR

44:22 Everysight

Everysight has AR glasses in a glasses-like form factor. They are designed to be self-contained, weigh only 47 grams, and have no external wiring. They use a 640×400 pixel full-color OLED display and can achieve >1000 nits to the eye.

Everysight uses a “Pre-Compensated Off-Axis” optical design, which tends to get more than double the light from the display to the eye while enabling more than three times the real-world light to pass through the display area compared to birdbath (e.g., Xreal) designs. With this design, the pre-compensation optics pre-correct for hitting the curved semi-mirror combiner off-axis. Typically, this mirror will be 50% or less reflective and only has to be applied over where the display is to be seen.

However, the Everysight glasses only support a rather small 22-degree FOV, and the eyebox is rather small. While Everysight has reduced the panoscopic tilt of the lenses over prior models, the latest Maverick modes still tilt toward the user’s cheeks more than most common glasses.

UPDATE 4/2/2024: Everysight responded to my original eyebox comment, “With respect to the eyebox, we take care of that with different sizes (Maverick today has two sizes – Medium and Large). The important part is that once you have the correct size, glass or eye movements won’t take you out of the eyebox. We believe that this is a much better tradeoff than a one-size-fits-all [with] low optical efficiency and enables you to use OLEDs in sunny days outdoors, even with clear visors.“

Thus far, Everysight seems to be marketing its glasses more to the sports market, which needs s, lightight headsets with bright displays for outdoor use.

If vision correction is not required, the lenses can be easily swapped out for various types of tint. More recently, Everysight has been able to support prescription lenses. For prescriptions, the inner curved mirror corrects for the virtual image, and a corrective lens on the outside corrects for the real world, including correcting for the curvature of the inner surface with the semi-mirror.

Everysight spun out of the large military company ELBIT, which perfected the pre-compensated off-axis design for larger headsets. This optical design is famously used in the F35 helmet and, more recently, in the civilian aircraft Skylens head-wearable HUD display, which has received FAA approval for use in multiple civilian aircraft, including recently the 737ng family.

Everysight was discussed in my CES 2024 coverage linked to below:

• (2024) CES (Pt. 2), Sony XR, DigiLens, Vuzix, Solos, EverySight, Mojie, TCL color µLED

48:42 TCL RayNeo X2 and X2 Lite

At CES 2024, TCL showed their RayNex X2 and their newer X2 Light. I have worked with 3-chip LCOS projectors with an X-Cube in the past, and I was curious to see the image quality as I know from experience aligning to X-Cubes is non-trivial, particularly with the smaller sizes of the Jade Bird Display red, green, and blue MicroLED displays.

Overall, the newer X2 Lite using the Applied Materials (AMAT) waveguides looked much better than the earlier RayNeo X2 (non-Lite). Even the AMAT had significant front projection, but as discussed with respect to Displelix above, this problem can be managed, at least for smaller FOVs (the RayNeo X2s have a ~30-degree diagonal FOV).

I covered the TCL color µLED in more detail in my CES 2024 coverage (link below). I have also included links to articles discussing the Jade Bird Displays MicroLEDs and their use of an X-Cub for a color combiner:

• (2024) CES (Pt. 2), Sony XR, DigiLens, Vuzix, Solos, EverySight, Mojie, TCL color µLED
• AWE 2022 (Part 6) – MicroLED Microdisplays for Augmented Reality (27:43 Jade Bird MicroLED Color X-Cube)
• (2021) News: WaveOptics & Jade Bird Display MicroLED Partnership

55:54 Mojie/Meta Bounds

Mojie/Meta Bound showed 640×480 green-only MicroLED-based glasses claiming 3,000 nits (to the eye), 90% transparency (without tinting), a 28-degree FOV, and a weight of only 38 grams. These were also wireless and, to a first approximation, very similar to Vuzix UltraLite. One thing that makes them stand out is that they use a waveguide technology made of plastic resin (most use glass).

Many companies are experimenting with plastic waveguides to reduce weight and improve safety. So far, the color uniformity with full-color displays has been worse than with glass-based waveguides. However, the uniformity issues are less noticeable with a monochrome (green) display. Mitsui Chemicals and Mitsubishi Chemicals, both of Japan, are suppliers of resin plastic substrate material for waveguides.

Below is a link to my article on Mojie/Meta Bounds in my CES 2024 coverage:

• (2024) CES (Pt. 2), Sony XR, DigiLens, Vuzix, Solos, EverySight, Mojie, TCL color µLED

57:59 Canon Mixed Reality

Canon had a fun demo based on the 100+ camera Free Viewpoint Video System VR system. Basically, you could sit around a table and see a basketball game (I think it was the 2022 NBA All-Stars Game) played on that table from any angle. Canon has been working on this technology for a decade or more, with demos for both basketball and soccer (football). While it’s an interesting technology demo, I don’t see how this would be a great way to watch a complete game. Even with over 100 cameras and the players being relatively small (far away virtually), one could see gaps where that the cameras couldn’t cover.

Canon also showed a very small passthrough AR camera and lens setup. While it was small, the FOV and video quality were not impressive. Brad Lynch of SadlyItsBradley found it to be pointless.

I have personally purchased a lot of Canon camera equipment over the last 25 years (including my Canon R5, which I take pictures with for this blog), so I am not in any way against Canon. However, as I discussed with Brad Lynch about Canon’s booth at CES 2023 (YouTube Link), I can’t see where Canon is going or what message they are trying to send in terms of mixed reality despite their very large and expensive booth. On the surface, Canon seems to be dabbling in various MR technologies, but it is not moving in a clear direction.

59:54 Solos (and Audio Glasses)

Solos makes audio-only glasses similar to the Meta/RayBand Wayfarer (but without cameras). These glasses emphasize modular construction, with all the expensive “smarts” in the temples so that the front-part lenses can be easily swapped.

Like several others, Solos uses cellular communication to connect to ChatGPT to do on-the-fly translations. What makes Solos more interesting is that Its Chairman is John Fan, also the chairman of Lightning Silicon Technology (a spinoff of Kopin Displays), a maker of OLED Microdisplays. At Lighting Silicon’s CES 2024 suite, John Fan discussed that incorporating the displays into the Solos glasses was an obvious future step.

CES (Pt. 2), Sony XR, DigiLens, Vuzix, Solos, EverySight, Mojie, TCL color µLED

1:01:16 Xander

While I saw Xander in the AARP sponsor AgeTech Summit booth at CES 2024, I didn’t get to meet with them. Xander hits at a couple of issues I feel are important. First, they show how AR technology can be used to help people. Secondly, they show what is expected to be a growing trend of adding basic visual information to augment audio.

While I (Karl) missed Xander at CES 2024, it turns out that Jason McDowall’s The AR Show (with guest host Kaden Pierce) recently interviewed Xander CEO Alex Westner on The AR Show.

Next Time – Optics and Display Devices

The video’s next part will discuss optical and display device companies.

Introduction

Many media outlets, large and small, both text and video, use this blog as a resource for technical information on mixed reality headsets. Sometimes, they even give credit. In the past two weeks, this blog was prominently cited in YouTube videos by Linus Tech Tips (LTT) and Artur Tech Tales. Less fortunately, Adam Savage’s Tested, hosted by Norman Chen in his Apple Vision Pro Review, used a spreadsheet test pattern from this blog to demonstrate foveated rendering issues.

I will follow up with a discussion of Linus’s Tech Tips video, which deals primarily with human factors. In particular, I want to discuss the “Information Density issue” of virtual versus physical monitors, which the LTT video touched on.

Influencing the Influencers On Apple Vision Pro

Linus Tech Tips (LTT)

In their “Apple Vision Pro—A PC Guy’s Perspective,” Linus Tech Tips showed several pages from this blog that were nice enough to prominently feature the pages they were using and the web addresses (below). Additionally, I enjoyed their somewhat humorous physical “simulation” of the AVP (more on that in a bit). LTT used images (below-left and below-center) from the blog to explain how the optics distort the display and how the processing in the AVP is used in combination with eye tracking to reduce that distortion. LTT also uses images from the blog (below-right) to show how the field of view (FOV) changes based on the distance from the eye to the optics.

Adam Savages’ Tested

Adam Savage’s Test with host Norman Chan’s review of the Apple Vision Pro used this blog’s AVP-XLS-on-BLACK-Large-Array from Spreadsheet “Breaks” The Apple Vision Pro’s (AVP) Eye-Tracking/Foveation & the First Through-the-optics Pictures to discuss how the foveated boundaries of the Apple Vision Pro are visible. While the spreadsheet is taken from this blog, I didn’t see any references given.

The Adam Savages Tested video either missed or was incorrect on several points it made:

• It missed the point of the blog article that the foveated rendering has problems with spreadsheets when directly rendered from Excel on the AVP instead of mirrored by a MacBook.
• It stated that taking pictures through the optics is impossible, which this blog has been doing for over a month (including in this article).
• It said that the AVP’s passthrough 3-D perspective was good with short-range but bad with long-range objects, but Linuses Tech tips (discussed later) find the opposite. The AVP’s accuracy is poor with short-range objects due to the camera placement.
• It said there was no “warping” of the real world with video passthrough, which is untrue. The AVP does less warping than the Meta Quest 3 and Quest Pro, but it still warps objects less than 0.6 meters (2 feet) away and toward the center to the upper part of the user’s view. It is impossible to be both perspective-correct and not warp with the AVP’s camera placement with near objects; the AVP seems to trade off being perspective-correct to have less warping than the Meta headsets.

Artur’s Tech Tales – Interview on AVP’s Optical Design

Artur’s Tech Tales Apple Vision Pro OPTICS—Deep Technical Analysis, featuring Arthur Rabner (CEO of Hypervision), includes an interview and presentation by Hypervision’s CEO, Arther Rabner. In his presentation, Rabner mentions this blog several times. The video details the AVP optics and follows up on Hypervision’s white paper discussed in Apple Vision Pro (Part 4) – Hypervision Pancake Optics Analysis.

Linus Tech Tips on Apple Vision Pro’s Human Factors

Much of the Linus Tech Tips (LTT) videos deal with human factors and user interface issues. For the rest of this article, I will discuss and expand upon comments made in the LTT video. Linus also commented on the passthrough camera’s “shutter angle,” but I moved my discussion on that subject to the “Appendix” at the end as it was a bit off-topic and needed some explanation.

It makes a mess of your face

At 5:18 in the video, Linus takes the headset off and shows the red marks left by the Apple Vision Pro (left), which I think may have been intentional after Linus complained about issues with the headband earlier. For reference, I have included the marks left by the Apple Vision Pro on my face (below-right). I sometimes joke that I wonder if I wear it long enough, it will make a groove in my skull to help hold up the headset.

An Apple person who is an expert at AVP fitting will probably be able to tell based on the marks on our faces if we have the “wrong” face interface. Linus’s headset makes stronger marks on his cheeks, whereas mine makes the darkest marks on my forehead. As I use inserts, I have a fairly thick (but typical for wearing inserts) 25W face hood with the thinner “W” interface, and AVP’s eye detection often complains that I need to get my eyes closer to the lenses. So, I end up cranking the solo band almost to the point where I feel my pulse on my forehead like a blood pressure measuring cuff (perhaps a health “feature” in the future?).

Need for game controllers

For virtual reality, Linus is happy with the resolution and placement of virtual objects in the real world. But he stated, “Unfortunately, the whole thing falls apart when you interact with the game.” Linus then goes into the many problems of not having controllers and relying on hand tracking alone.

I’m not a VR gamer, but I agree with The Verge that AVP’s hand and eye tracking is “magic until it’s not.” I am endlessly frustrated with eye-tracking-based finger selection. Even with the headset cranked hard against my face, the eye tracking is unstable even after recalibration of the IPD and eye tracking many times. I consider eye and hand tracking a good “secondary” selection tool that needs an accurate primary selection tool. I have an Apple Magic Pad that “works” with the AVP but does not work in “3-D space.”

Windows PC Gaming Video Mirroring via WiFi has Lag, Low Resolution, and Compression Artifacts

Linus discussed using the Steam App on the AVP to play games. He liked that he could get a large image and lay back, but there is some lag, which could be problematic for some games, particularly competitive ones; the resolution is limited to 1080p, and compression artifacts are noticeable.

Linus also discussed using the Sunshine (streaming server on the PC) and Moonlight (remote access on the AVP) apps to mirror Windows PCs. While this combination supports up to 4K at 120p, Linus says you will need an incredibly good wireless access point for the higher resolution and frame rates. In terms of effective resolution and what I like to call “Information Density,” these apps will still suffer the loss of significant resolution due to trying to simulate a virtual monitor in 3-D space, as I have discussed in Apple Vision Pro (Part 5C) – More on Monitor Replacement is Ridiculous and Apple Vision Pro (Part 5A) – Why Monitor Replacement is Ridiculous and shown with through the lens pictures in Apple Vision Pro’s (AVP) Image Quality Issues – First Impressions and Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3.

From a “pro” design perspective, it is rather poor on Apple’s part that the AVP does not support a direct Thunderbolt link for both data and power, while at the same time, it requires a wired battery. I should note that the $300 developer’s strap supports a lowish 100Mbs ethernet (compared to USB-C/Thunderbolt 0.48 to 40 Gbs) speed data through a USB-C connector while still requiring the battery pack for power. There are many unused pins on the developer’s strap, and there are indications in the AVP’s software that the strap might support higher-speed connections (and maybe access to peripherals) in the future.

Warping effect of passthrough

In terms of video passthrough, at 13:43 in the video, Linus comments about the warping effect of close objects and depth perception being “a bit off.” He also discussed that you are looking at the world through phone-type cameras. When you move your head, the passthrough looks duller, with a significant blur (“Jello”).

The same Linus Tech Tip video also included humorous simulations of the AVP environment with people carrying large-screen monitors. At one point (shown below), they show a person wearing a respirator mask (to “simulate” the headset) surrounded by three very large monitors/TVs. They show how the user has to move their head around to see everything. LTT doesn’t mention that those monitors’ angular resolution is fairly low, which is why those monitors need to be so big.

Sharing documents is a pain.

Linus discussed the AVP’s difficulty sharing documents with others in the same room. Part of this is because the MacBook’s display goes blank when mirroring onto the AVP. Linus discussed how he had to use a “bizarre workaround” of setting up a video conference to share a document with people in the same room.

Information Density – The AVP Delivers Effectively Multiple Large but Very Low-Resolution Monitors

The most important demonstration in the LTT video involves what I like to call the “Information Density” problem. The AVP, or any VR headset, has low information density when trying to emulate a 2-D physical monitor in 3-D space. It is a fundamental problem; the effective resolution of the AVP well less than half (linearly, less than a quarter two-dimensionally) of the resolution of the monitors that are being simulated (as discussed in Apple Vision Pro (Part 5C) – More on Monitor Replacement is Ridiculous and Apple Vision Pro (Part 5A) and shown with through the lens pictures in Apple Vision Pro’s (AVP) Image Quality Issues – First Impressions and Apple Vision Pro’s Optics Blurrier & Lower Contrast than Meta Quest 3). The key contributors to this issue are:

• The peak display resolution in the center of the optics is only 44.4 pixels per degree (human vision it typically better than 60 ppd).
• The 2-D/Monitor image must be resampled into 3-D space with an effective resolution loss greater than 2x.
• If the monitor is to be viewable, it must be inscribed inside the oval sweet spot of the optics. In the case of the AVP, this cuts off about half the pixels.
• While the AVP’s approximate horizontal FOV is about 100 degrees, the optical resolution drops considerably in the outer third of the optics. Only about the center 40-50 degrees of the FOV is usable for high-resolution content.
• Simply put, the AVP needs more than double the PPD and better optics to provide typical modern computer monitors’ information/resolution density. Even then, it would be somewhat lacking in some aspects.

Below, show the close-up center (best case) through the AVP’s optics on the (left) and the same image at about the same FOV on a computer monitor (right). Things must be blown up about 2x (linearly) to be as legible on the AVP as on a good computer monitor.

Some current issues with monitor simulation are “temporary software issues” that can be improved, but that is not true with the information density problem.

Linus states in the video (at 17:48) that setting up the AVP is a “bit of a chore,” but it should be understood most of the “chore” is due to current software limitations that could be fixed with better software. The most obvious problems, as identified by Linus, are that the AVP does not currently support multiple screens from a MacBook, and it does not save the virtual screen location of the MacBook. I think most people expect Apple to fix these problems at some point in the near future.

At 18:20, Linus showed the real multiple-monitor workspace of someone doing video editing (see below). While a bit extreme for some people with two vertically stacked 4K monitors in landscape orientation monitors and a third 4K monitor in portrait mode, it is not that far off what I have been using for over a decade with two large side-by-side monitors (today I have a 34″ 22:9 1440p “center monitor” and a 28″ 4K side monitor both in landscape mode).

I want to note a comment made by Linus (with my bold emphasis):

“Vision Pro Sounds like having your own personal Colin holding a TV for you and then allowing it to be repositioned and float effortlessly wherever you want. But in practice, I just don’t really often need to do that, and neither do a lot of people. For example, Nicole, here’s a real person doing real work [and] for a fraction of the cost of a Vision Pro, she has multiple 4K displays all within her field of view at once, and this is how much she has to move her head in order to look between them. Wow.  

Again, I appreciate this thing for the technological Marvel that it is—a 4K display in a single Square inch. But for optimal text clarity, you need to use most of those pixels, meaning that the virtual monitor needs to be absolutely massive for the Vision Pro to really shine.“

The bold highlights above make the point about information density. A person can see all the information all at once and then, with minimal eye and head movement, see the specific information they want to see at that moment. Making text bigger only “works” for small amounts of content as it makes reading slower with larger head and eye movement and will tend to make the eyes more tired with movement over wider angles.

To drive the point home, the LTT video “simulates” an AVP desktop, assuming multiple monitor support but physically placing three very large monitors side by side with two smaller displays on top. They had the simulated user wear a paint respirator mask to “simulate” the headset (and likely for comic effect). I would like to add that each of those large monitors, even at that size, with the AVP, will have the resolution capability of more like a 1920x1080p monitor or about half linearly and one-fourth in area, the content of a 4K monitor.

Quoting Linus about this part of the video (with my bold emphasis):

It’s more like having a much larger TV that is quite a bit farther away, and that is a good thing in the sense that you’ll be focusing more than a few feet in front of you. But I still found that in spite of this, that it was a big problem for me if I spent more than an hour or so in spatial-computing-land.

Making this productivity problem worse is the fact that, at this time, the Vision Pro doesn’t allow you to save your layouts. So every time you want to get back into it, you’ve got to put it on, authenticate, connect to your MacBook, resize that display, open a safari window, put that over there where you want it, maybe your emails go over here, it’s a lot of friction that our editors, for example, don’t go through every time they want to sit down and get a couple hours of work done before their eyes and face hurt too much to continue.

I would classify Many of the issues Linus gave in the above quote as solvable in software for the AVP. What is not likely solvable in software are headaches, eye strain, and low angular resolution of the AVP relative to a modern computer monitor in a typical setup.

While speaking in the Los Angeles area at the SID LA One Day conference, I stopped in a Bigscreen Beyond to try out their headset for about three hours. I could wear the Bigscreen Beyond for almost three hours, where typically, I get a spitting headache with the AVP after about 40 minutes. I don’t know why, but it is likely a combination of much less pressure on my forehead and something to do with the optics. Whatever it is, there is clearly a big difference to me. It was also much easier to drink from a can (right) with the Bigscreen’s much-reduced headset.

Conclusion

It is gratifying to see the blog’s work reach a wide audience worldwide (about 50% of this blog’s audience is outside the USA). As a result of other media outlets picking up this blog’s work, the readership roughly doubled last month to about 50,000 (Google Analytics “Users”).

I particularly appreciated the Linus Tech Tip example of a real workspace in contrast to their “simulation” of the AVP workspace. It helps illustrate some human factor issues with having a headset simulate a computer monitor, including information density. I keep pounding on the Information Density issue because it seems underappreciated by many of the media reports on the AVP.

Appendix Linus Comments on AVP’s “Weird Camera Shutter Angle”

I moved this discussion to this Appendix because it involves some technical discussion that, while it may be important, may not be of interest to everyone and takes some time to explain. At the same time, I didn’t want to ignore it as it brings up a potential issue with the AVP.

At about 16:30 in the LTT Video, Linus also states that the Apple Vision Pro cameras use “weird shutter angles to compensate for the flickering of lights around you, causing them [the AVP] to crank up the ISO [sensitivity], adding a bunch of noise to the image.”

For those that don’t know, “shutter angle” (see also https://www.digitalcameraworld.com/features/cheat-sheet-shutter-angles-vs-shutter-speeds) is a hold-over term from the days of mechanical movie shutters where the shutter was open for a percentage of a 360-degree rotating shutter (right). Still, it is now applied to camera shutters, including “electronic shutters” (many large mirrorless cameras have mechanical and electronic shutter options with different effects). A 180-degree shutter angle means the shutter/camera scanning is open one-half the frame time, say 1/48th of a 1/24th of a second frame time or 1/180th of a 1/90th of a second frame rate. Typically, people talk about how different shutter angles affect the choppiness of motion and motion blur, not brightness or ISO, even though it does affect ISO/Brightness due to the change in exposure time.

I’m not sure why Linus is saying that certain lights are reducing the shutter angle, thus increasing ISO, unless he is saying that the shutter time is being reduced with certain types of light (or simply bright lights) or with certain types of flickering lights the cameras are missing much of the light. If so, it is a roundabout way of discussing the camera issue; as discussed above, the term shutter angle is typically used in the context of motion effects, with brightness/ISO being more of a side issue.

A related temporal issue is the duty cycle of the displays (as opposed to the passthrough cameras), which has a similar “shutter angle” issue. VR users have found that displays with long on-time duty cycles cause perceived blurriness with rapid head movement. Thus, they tend to prefer display technologies with low-duty cycles. However, low display duty cycles typically result in less display brightness. LED backlit LCDs can drive the LEDs harder for shorter periods to help make up for the brightness loss. However, OLED microdisplays commonly have relatively long (sometimes 100%) on-time duty cycles. I have not yet had a chance to check the duty cycle of the AVP, but it is one of the things on my to-do list. In light of Linus’s comments, I will want to set up some experiments to check out the temporal behavior of the AVP’s passthrough camera.

Introduction – Sorry, But It’s True

I have taken thousands of pictures through dozens of different headsets, and I noticed that the Apple Vision Pro (AVP) image is a little blurry, so I decided to investigate. Following up on my Apple Vision Pro’s (AVP) Image Quality Issues – First Impressions article, this article will compare the AVP to the Meta Quest 3 by taking the same image at the same size in both headsets, and I got what many will find to be surprising results.

I know all “instant experts” are singing the praises of “the Vision Pro as having such high resolution that there is no screen door effect,” but they don’t seem to understand that the screen door effect is hiding in plain sight, or should I say “blurry sight.” As mentioned last time, the AVP covers its lower-than-human vision angular resolution by making everything bigger and bolder (defaults, even for the small window mode setting, are pretty large).

While I’m causing controversies by showing evidence, I might as well point out that the AVP’s contrast and color uniformity are also slightly lower than the Meta Quest 3 on anything but a nearly black image. This is because the issues with AVP’s pancake optics dominate over AVP’s OLED microdisplay. This should not be a surprise. Many people have reported “glow” coming from the AVP, particularly when watching movies. That “glow” is caused by unwanted reflections in the pancake optics.

If you click on any image in this article, you can access it in full resolution as cropped from a 45-megapixel original image. The source image is on this blog’s Test Pattern Page. As if the usual practice of this blog, I will show my work below. If you disagree, please show your evidence.

Hiding the Screen Door Effect in Plain Sight with Blur

The numbers don’t lie. As I reported last time in Apple Vision Pro’s (AVP) Image Quality Issues – First Impressions, the AVP’s peak center resolution is about 44.4 pixels per degree (PPD), below 80 PPD, what Apple calls “retinal resolution,” and the pixel jaggies and screen door should be visible — if the optics were sharp. So why are so many reporting that the AVP’s resolution must be high since they don’t see the screen door effect? Well, because they are ignoring the issue of the sharpness of the optics.

Two factors affect the effective resolution: the PPD of optics and the optics’ modulation transfer function sharpness and contrast of the optics, commonly measured by the Modulation Transfer Function (MTF — see Appendix on MTF).

People do not see the screen door effect with the AVP because the display is slightly out of focus/blurry. Low pass filtering/blurring is the classic way to reduce aliasing and screen door effects. I noticed that when playing with the AVP’s optics, the optics have to be almost touching the display to be in focus. The AVP’s panel appears to be recessed by about 1 millimeter (roughly judging by my eye) beyond the best focus distance. This is just enough so that the thinner gaps between pixels are out of focus while only making the pixels slightly blurry. There are potentially other explanations for the blur, including the microlenses over the OLED panel or possibly a softening film on top of the panel. Still, the focus seems to be the most likely cause of the blurring.

Full Image Pictures from the center 46 Degrees of the FOV

I’m going to start with high-resolution pictures through the optics. You won’t be able to see any detail without clicking on them to see them at full resolution, but you may discern that the MQ3 feels sharper by looking at the progressively smaller fonts. This is true even in the center of the optics (square “34” below), even before the AVP’s foveate rendering results in a very large blur at the outside of the image (11, 21, 31, 41, 51, and 61). Later, I will show a series of crops to show the central regions next to each other in more detail.

The pictures below were taken by a Canon R5 (45 Megapixel) camera with a 16mm lens at f8. With a combination of window sizing and moving the headset, I created the same size image on the Apple Vision Pro and Meta Quest Pro to give a fair comparison (yes, it took a lot of time). A MacBook Pro M3 Pro was casting the AVP image, and the Meta Quest 3 was running the Immersed application (to get a flat image) mirroring a PC laptop. For reference, I added a picture of a 28″ LCD monitor taken from about 30″ to give approximately the same FOV as the image from a conventional 4K monitor (this monitor could resolve single pixels of four of these 1080p images, although you would have to have very good vision see them distinctly).

Medium Close-Up Comparison

Below are crops from near the center of the AVP image (left), the 28″ monitor (center), and the MQ3 image (right). The red circle on the AVP image over the number 34 is from the eye-tracking pointer being on (also used to help align and focus the camera). The blur of the AVP is more evident in the larger view.

Extreme Close-Up of AVP and MQ3

Cropping even closer to see the details (all the images above are at the same resolution) with the AVP on the top and the MQ3 on the bottom. Some things to note:

1. Neither the AVP nor MQ3 can resolve the 1-pixel lines, even though a cheap 1080p monitor would show them distinctly.
2. While the MQ3 has more jaggies and the screen door effect, it is noticeably sharper.
3. Looking at the space between the circle and the 3-pixel wide lines pointed at by the red arrow, it should be noticed that the AVP has less contrast (is less black) than the MQ3.
4. Neither the AVP nor MQ3 can resolve the 1-pixel-wide lines correctly, but the 2- and 3-pixel-wide lines, along with all the text, are significantly sharper and have higher contrast than on the AVP. Yes, the effective resolution of the MQ3 is objectively better than the AVP.
5. Some color moiré can be seen in the MQ3 image, a color artifact due to the camera’s Bayer filter (not seen by the eye) and the relative sharpness of the MQ3 optics. The camera can “see” the MQ3’s LCD color filters through the optics.

Experiment with Slightly Blurring the Meta Quest 3

A natural question is whether the MQ3 should have made their optics slightly out of focus to hide the screen door effect. As a quick experiment, I tried a (Gaussian) blur of the MQ3’s image a little (middle image below) as an experiment. There is room to blur it while still having a higher effective resolution than the AVP. The AVP still has more pixels, and the person/elf’s image looks softer on the slightly blurred MQ3. The lines are testing for high contrast resolution (and optical reflections), and the photograph shows what happens to a lower contrast, more natural image with more pixel detail.

AVP’s Issues with High-Resolution Content

While Apple markets each display as having the same number of pixels as a 4K monitor (but differently shaped and not as wide), the resolution is reduced by multiple factors, including those listed below:

1. The oval-shaped optics cut about 25-30% of the pixels.
2. The outer part of the optics has poor resolution (about 1/3rd the pixels per degree of the center) and has poor color.
3. A rectangular image must be inscribed inside the “good” part of the oval-shaped optics with a margin to support head movement. While the combined display might have a ~100-degree FOV, there is only about a 45- to 50-degree sweet spot.
4. Any pixels in the source image must be scaled and mapped into the destination pixels. For any high-resolution content, this can cause more than a 2x (linear) loss in resolution and much worse if it aliases. For more on the scaling issues, see my articles on Apple Vision Pro (Part 5A, 5B, & 5C).
5. As part of #4 above or in a separate process, the image must be corrected for optical distortion and color as a function of eye tracking, causing further image degradation
6. Scintillation and wiggling of high-resolution content with any head movement.
7. Blurring by the optics

The net of the above, and as demonstrated by the photographs through the optics shown earlier, the AVP can’t accurately display a detailed 1920×1080 (1080p) image.

AVP Lack “Information Density”

Making everything bigger, including short messages and videos, can work for low-information-density applications. If anything, the AVP demonstrates that very high resolution is less important for movies than people think (watching movies is a notoriously bad way to judge resolution).

As discussed last time, the AVP makes up the less-than-human angular resolution by making everything big to hide the issue. But making the individual elements bigger means less content can be seen simultaneously as the overall image is enlarged. But making things bigger means that the “information density” goes down, with the eyes and head having to move more to see the same amount of content and less overall content can be seen simultaneously. Consider a spreadsheet; fewer rows and columns will be in the sweet spot of a person’s vision, and less of the spreadsheet will be visible without needing to turn your head.

This blog’s article, FOV Obsession, discusses the issue of eye movement and fatigue using information from Thad Starner’s 2019 Photonic’s West AR/VR/MR presentation. The key point is that the eye does not normally want to move more than 10 degrees for an extended period. The graph below left is for a monocular display where the text does not move with the head-turning. Starner points out that a typical newspaper column is only about 6.6 degrees. It is also well known that when reading content more than ~30 degrees wide, even for a short period, people will turn their heads rather than move their eyes. Making text content bigger to make it legible will necessitate more eye and head movement to see/read the same amount of content, likely leading to fatigue (I would like to see a study of this issue).

ANSI-Like Contrast

A standard way to measure contrast is using a black-and-white checkerboard pattern, often called ANSI Contrast. It turns out that with a large checkerboard pattern, the AVP and MQ3 have very similar contrast ratios. For the picture below, I make the checkerboard bigger to fill about 70 degrees horizontally for each device’s FOV. The optical reflections inside the AVP’s optics cancel out the inherent high contrast of the OLED displays inside the AVP.

The AVP Has Worse Color Uniformity than the MQ3

You may be able to tell that the AVP has a slightly pink color in the center white squares. As I move my head around, I see the pink region move with it. Part of the AVP’s processing is used to correct color based on eye tracking. Most of the time, the AVP does an OK job, but it can’t perfectly correct for color issues with the optics, which becomes apparent in large white areas. The issues are most apparent with head and eye movement. Sometimes, by Apple’s admission, the correction can go terribly wrong if it has problems with eye tracking.

Using the same images above and increasing the color saturation in both images by the same amount makes the color issues more apparent. The MQ3 color uniformity only slightly changes in the color of the whites, but the AVP turns pink in the center and cyan on the outside.

The AVP’s “aggressive” optical design has about 1.6x the magnification of the MQ3 and, as discussed last time, has a curved quarter waveplate (QWP). Waveplates modify polarized light and are wavelength (color) and angle of light-dependent. Having repeatedly switched between the AVP and MQ3, the MQ3 has better color uniformity, particularly when taking one off and quickly putting the other on.

Conclusion and Comments

As a complete product (more on this in future articles), the AVP is superior to the Meta Quest Pro, Quest 3, or any other passthrough mixed reality headset. Still, the AVP’s effective resolution is less than the pixel differences would suggest due to the softer/blurrier optics.

While the pixel resolution is better than the Quest Pro and Quest 3, its effective resolution after the optics is worse on high-contrast images. Due to having a somewhat higher PPD, the AVP looks better than the MQP and MQ3 on “natural” lower-contrast content. The AVP image is much worse than a cheap monitor displaying high-resolution, high-contrast content. Effectively, what the AVP supports is multiple low angular resolution monitors.

And before anyone makes me out to be a Meta fanboy, please read my series of articles on the Meta Quest Pro. I’m not saying the MQ3 is better than the AVP. I am saying that the MQ3 is objectively sharper and has better color uniformity. Apple and Meta don’t get different physics, and they make different trade-offs which I am pointing out.

The AVP and any VR/MR headset will fare much better with “movie” and video content with few high-contrast edges; most “natural” content is also low in detail and pixel-to-pixel contrast (and why compression works so well with pictures and movies). I must also caution that we are still in the “wild enthusiasm stage,” where the everyday problems with technology get overlooked.

In the best case, the AVP in the center of the display gives the user a ~20/30 vision view of its direct (non-passthrough) content and worse when using passthrough (20/35 to 20/50). Certainly, some people will find the AVP useful. But it is still a technogeek toy. It will impress people the way 3-D movies did over a decade ago. As a reminder, 3-D TV peaked at 41.45 million units in 2012 before disappearing a few years later.

Making a headset display is like n-dimensional chess; more than 20 major factors must be improved, and improving one typically worsens other factors. These factors include higher resolution, wider FOV, peripheral vision and safety issues, lower power, smaller, less weight, better optics, better cameras, more cameras and sensors, and so on. And people want all these improvements while drastically reducing the cost. I think too much is being made about the cost, as the AVP is about right regarding the cost for a new technology when adjusted for inflation; I’m worried about the other 20 problems that must be fixed to have a mass-market product.

Appendix – Modulation Transfer Function (MTF)

MTF is measured by putting in a series of lines of equal width and spacing and measuring the difference between the white and black as the size and spacing of the lines change. People typically use 50% contrast critical to specify the MTF by convention. But note that contrast is defined as (Imax-Imin)/(Imax+Imin), so to achieve 50% contrast, the black level must be 1/3rd of the white level. The figure (below) shows how the response changes with the line spacing.

The MTF of the optics is reduced by both the sharpness of the optics and any internal reflections that, in turn, reduce contrast.

Introduction

Today’s article is just some early findings on the Apple Vision Pro (AVP). I’m working on many things related to the AVP, and it will take me a while to prepare all of them for publishing. Among the things I am doing, I am trying to capture “through the optics” pictures of the AVP, and it is unveiling both interesting information on how the AVP works and the second test pattern I tried “broke” the foveated rending of the AVP.

Having done several searches, I have not seen any “through-the-optics” pictures of the AVP yet. They may be out there, but I haven’t found them. So, I thought I would put up a couple of my test pictures to be (hopefully) the first to publish a picture through the AVP’s optics.

Eye Tracking Display Based Rendering, Maybe “Too smart for its own good”

The AVP is sometimes “too smart for its own good,” resulting in bad visual artifacts. In addition to being used for selection, the AVP’s eye-tracking varies the resolution and corrects color issues (aberrations) in the optics by pre-processing the image. This makes it tricky to photograph because the camera lens looks different to the human eye.

Today, I threw together some spreadsheets to check my ability to take pictures through the AVP optics. I started with two large Excel spreadsheets displayed using the AVP’s native Excel App. One spreadsheet used black text on a white background, which looked like the AVP was making the text and lines look “bolder/thicker” than they should look, but it didn’t act that crazy; the AVP seems to be “enhancing” (not always what you want) the spreadsheet’s readability.

But then I tried inverting everything with white text and lines on a black background, and the display started scintillating in a square box that followed the eye tracking. Fortunately, the AVP’s recording captured the effect in the video below.

I want to emphasize that it is not just the camera or the AVP’s video capture that shows the problem with the foveated rendering; I see it with my own eyes. I have provided the spreadsheets below so anyone with an AVP can verify my findings. I have only tested this with the Excel running on the AVP. The effect is most easily seen if you go into “View” in Excel” and make the view smaller with “-” magnifying glass 3 or 4 times to make the text and boxes smaller.

My First Through-the-Optics Picture Experiments

With its eye-tracking-based rendering, the AVP will be tricky to capture through the optics. The tracking behaves differently with different cameras and lenses. When setting up the camera, I can see the AVP changing colors, sometimes resulting in pictures that are colored differently than what my eye sees.

It seems pretty clear that the AVP is using “foveated,” variable resolution rendering even on still subjects like a spreadsheet. This re-rendering is based on the eyes and due to the change in the 3-D space locking (aka, SLAM) that caused the artifacts seen in the White text and lines on the BLACK spreadsheet.

Furthermore, the resolution of the displays is definitely lower than the eye’s resolution, as you can easily see the anti-aliasing “twisted rope” rippling effect if you look at the white-on-black spreadsheet. The highest rendered resolution (“foveated”) part of the image that scintillates. I discussed this issue in Apple Vision Pro (Part 5A) – Why Monitor Replacement is Ridiculous, Apple Vision Pro (Part 5B) – More on Monitor Replacement is Ridiculous, and Apple Vision Pro (Part 5C) – More on Monitor Replacement is Ridiculous.

I should point out that if not for the foveation, the whole image would scintillate. Still, the foveated rendering worsens because it creates a visible square at the boundary between the foveated area and the lower-resolution region. The “foveated rendering” makes it worse by changing the text and lines’ resolution and thickness. I would argue that a more graceful degradation would be to have the whole image rendered the same way (it is not a processing limitation to render a spreadsheet), with the whole image scintillating rather than having boundary lines where it does and does not and with the boldness changing at the boundaries as well. The key point is that the AVP’s display, while much better than almost all other VR/MR headsets, is not, as Apple puts it, “retinal resolution” (or beyond what the eye can see).

Anyway, for the record, below are a couple of through-the-optics test pictures. The first was taken with an R5 camera with a 28mm lens and “pixel shifting” to give a 400-megapixel image. Click on the crop of a very small portion of the center of that picture below to see it in full resolution.

The second image below was taken with an Olympus D mark III (Micro Four-Thirds camera) with a 17mm lens. It does not have the resolution of the R5, but the AVP’s eye tracking behaves better with this lens. This camera has a 24mp sensor, and then I used its pixel-shifting feature to capture the image at about 80 megapixels. The whole image (click to see at full resolution) is included below.

If you scroll around the full-resolution image, you can make out the pixel grid through most of the image, yet the text becomes blurrier much more quickly. Preliminarily, this seems to suggest foveated rendering. I have not had time to check yet, but I suspect the resolution falloff coincides with the squares in the white-on-black spreadsheet.

Conclusion

Designers have to be careful when it comes to applying technology. Sometimes, the same smarts that make one thing work will make others behave poorly.

The biggest part of the problem could be a bug in the AVP software or the Excel port. I’m not saying it is the end of the world, even if it is not improved. There is probably a way to “tone down” the foveated rending to reduce this problem, but I don’t think there is any way to eliminate it, given the display resolution. At the same time, the second test I tried caused it to “break/escape.” Since it happens so readily, this problem will likely show up elsewhere. Fundamentally, it comes down to the display not having a resolution as good as human vision.