Introducing Project HoloLens

Project HoloLens

Recently Microsoft unveiled technology at the center of a project seven years in development – a head-mounted holographic computer developed under the title Project HoloLens.

Intriguing, exciting and ambitious, Project HoloLens is pitched to deliver a whole new platform for interacting with digital media through novel holographic technology. The headset, HoloLens, features state-of-the-art technology that manipulates light to augment the user’s visual perception of reality by projecting holograms as features embedded in the real world.

While it was only a demonstration of a prototype for participants of the Windows 10 event (January 21) taking place at the company’s headquarters, the event’s presentation of HoloLens provided not only the first official look at the new kit, but also a tantalising glimpse into the future direction we might imagine Microsoft to be taking.

Fortunately, aside from their release of several nice promotional videos for the headset, Microsoft have granted members of the press hands-on demonstrations of their new gear; and as such there are some insightful testimonials out there (see Resources beneath) providing hints at what to expect when the product is launched to developers, perhaps as early as this spring.

 

Living up to expectations

Holograms – a staple technology of science fiction. It goes without saying that most of us already have an impression of what holograms would look like. Classically, it’s likely to involve some 3-dimensional projection of light into a volume of space, visible by naked eye, and manipulated carelessly with a wave of a hand.

Well, in the interests of being upfront – HoloLens doesn’t quite meet all these assumptions. HoloLens’ projections, aren’t really direct projections into the really world at all, as they can only be viewed through the lenses of the headset, as it’s this which houses the technology that creates an illusion of objects in the world beyond. However the translucency of the headset’s visor means that those hologram’s are perceived as embedded within our own 3-dimensional world (and not the mere few centimeters from your eyes in which they actually exist).

It ought to be made clear before delving into further description that what’s been on show to date is strictly prototype technology. The field of view of this beta-HoloLens has been described as tiny – a narrow band within which the real world fades, giving way to digital projections. Outside of this band the normal world remains; meaning that a considerable amount of head turning is required to survey a room for any and all projections on embedded within in. Embedded within it? As in, fixed in a position and awaiting an audience? Yes – but we’ll come to that…

At the same time the prototype’s physical architecture is still in development. For now, much of the hardware is housed in a case that hangs around the user’s neck; while the headset itself remains open to the elements, with circuitry and sensors visible.

Nevertheless, all good things in time – Microsoft have given assurances that the headset itself will eventually house all processing and power needs; and, in turn, that the device will be wireless. Equally, it’s a fair bet that the field of view will increase further too. But even with the current concessions the prevailing verdict from those fortunate enough to have trialed the prototype HoloLens is one of resounding collective approval. Project HoloLens is clearly onto something good.

Holograms, virtual environments and augmented reality

First things first – Project HoloLens isn’t just about holograms. It appears to be a system capable of creating, maintaining and providing a entirely new type of digital ecosystem for working, playing, communicating and engaging in. To be sure, HoloLens’ production of holograms is original: bright, bold, crisp projections to hear of it from those who’ve seen them. But it’s the way in which the technology is being applied that really is especially ambitious.

A critical feature of HoloLens is that it builds and maintains an ambient awareness of its surroundings. HoloLens actually scans and maps its environment, constructing a digital model in real time which is then used as a canvas onto which holograms can be projected. In this way, projections – be them animated characters, buildings, or whole landscapes – can be mapped onto the existing features of the environment.

Project HoloLens Augment

All this mapping is achieved using a large array of sensors, range finders, and cameras…exactly how it’s all working together, Microsoft haven’t said; but it seems to be functioning.

Featuring as one of their showcase demos for Project HoloLens, Microsoft have shown off a HoloLens-compatible version of Minecraft. (For those unfamiliar with Minecraft, think of it as Lego of the digital world) Testimonials tell of building houses on coffee tables, farms on floors and blasting through living room walls to reveal caverns behind…Project HoloLens Minecraft

It’s a neat demo, largely because it shows off how the mapping technology can be applied to bring about new experiences of engaging with digital media, but also because it appeals to an inherent appreciation for bringing creative products of a virtual world out of computers and screens, and into the real world.

Mapping environments and enveloping them within virtual projections is probably one of the most exciting aspects to HoloLens; not only does it open up a trove of applications – think design, modeling, and simulation – but it really brings about an unprecedented platform of inter-weaving between the digital and the real world.

Unlike entering an entirely virtual world, cut off from the world Oculus Rift-style, the wearer remains aware and a part of their actual surroundings. While with Oculus the goal is displacement and immersion into a virtual world – leading to the already fabled sense of existential ‘presence’ –, with HoloLens, displacement isn’t the objective at all. Rather, Microsoft are intent on bringing the digital world to you; or more precisely, superimposing the digital onto the real world.

You’ll be able to work at a computer whilst wearing HoloLens: looking over toward floating spreadsheets, data representations, images or designs, which hang off to one side. Since objects in the real world can be registered by HoloLens’ sensors, they can be used as foundations for projections. It’s this merging of the digital and the real that’s genuinely intriguing – and in all likelihood in the future for computing.

Another important aspect of HoloLens is its ability to lock holograms into position – something being referred to as ‘pinning’ by the project’s engineers. A user can pin a projection into some desired location, and then move around to inspect, or interact with the projection from any angle. A pinned object will presumably adopt a where-you-left-it property; locked into position, rather than moving relative to the user.

Just to underscore the value of this – imagine the scenario pictured below: pinning a globe. You’d be able to walk around it; set it spinning; adjust its size; get in for a closer look; and add additional objects to orbit around it.

Project HoloLens globe

Walking on Another Planet

Combining features of pinning with virtual landscaping leads to some exciting applications. One of the demos put together by Microsoft featured the red surface of Mars; users could walk around, inspect a looming Curiosity rover, plant flags, and mark areas for later inspection. The detail was supposedly terrific, and to add just a little more kudos to the demo, the data used in the simulation was actually imagery sent back by the NASA Mars rover, Curiosity. It’s certainly neat to imagine mapping out a forest floor, city street or shores of a lake, all on in your living room.

Project HoloLens Mars

Shared Experiences

Another notable demonstration from Microsoft came in the form of a Skype call featuring real-time annotations being made by a second-party that appeared in the field of view of the HoloLens user. Not only would HoloLens allow for a video of this second party to float nearby, but that person can also be granted a first-person perspective of what the user is seeing through HoloLens – ushering in a unique capacity for sharing experiences and perspectives. In a Skype-like scenario put together by Project HoloLens, a helpful second-party guided intre­pid journalists through the various steps of fixing a broken electrical socket.

 Project HoloLens Skyoe

Interfacing

HoloLens is spartan when it comes to a physical interface – in fact there are only manual controls for adjusting the volume and contrast, and a power switch. All other controls are dependent on gesture, voice and other information collected by HoloLens’ array of sensors. What’s clear therefore is that Project HoloLens is not simply about testing holographic displays; but also about pushing the boundaries of gesture based interfacing across a new platform.

Project HoloLens desktop

Gesture-based movements, of the head, hand and fingers, are central to controlling HoloLens. Judging from videos, testimonials and information released so far, gestures are allowing for grabbing holograms, clicking and selecting, as well as switching between views. It remains to be seen how much control will be afforded by gestures. Seemingly, the only real limitation is the capability of software such that it allows for recognition of more complex gestures.

One thing’s for sure though – the user’s sense of immediacy in being able to intuitively navigate through holographic spaces and features will be paramount to the success of the technology. Since a keyboard input isn’t being planned as an option for the HoloLens, Microsoft face the challenge of engineering this new formula interface to work right, first time.

In the mean time, Microsoft’s Holo Studio app nicely showed off how hand movements worked with HoloLens: supporting digital sculpting and design of objects, that are immediately ready for 3D printing.

Additional to the headset’s sensors are cameras that track eye movement. This allows for projections to be modulated by the direction one is looking. It also opens up avenues of opportunity for using gaze direction and points of focus for intuitive, smart interfacing – if the system knows where you are looking, not only can that holo-feature be brought into sharper focus, but it may be pre-emptively selected for interaction.

 

Inner Workings of HoloLens

In ways Project HoloLens may be a considered a successor to the motion-sensing gaming device Kinect, released in 2010 for Microsoft’s Xbox. But such a description doesn’t do HoloLens justice – the motion detection technology may hail from Kinect, but it’s vastly evolved, and the headset is obviously boasting features that go far beyond simply detecting the movement and position of a gamer.

We don’t know a great deal about precisely how HoloLens is working, but we have some information to be working with.

HoloLens creates the illusion of three dimensional objects by projecting light through its two lenses into each eye. Each lens incorporates three layers of glass – red, green and blue – each of which is filled with micro-thin corrugated grooves that diffract light. A so-called ‘light engine’ housed above the lenses projects (presumably in a highly complex fashion) light into the layers of glass where it bounces around between each of the layers to create holographic images that the mind interprets as solid objects located in the real world at varying depths.

 

 

The matter of how HoloLens models its environment is also curious, and not something we know a lot about. We know that are at least 18 sensors feeding information into HoloLens; presumably detecting depth, surfaces, and movement. The HoloLens homepage also notes that the device can detect composition of materials – hard versus soft for instance –, so again it’s hard to extrapolate on the full extent of capability.

HoloLen’s cameras have a field of view of 120 by 120 degree – considerably more than its antecedent Kinect technology – meaning that they can track movements from arms close to stretched out along both horizontal and vertical planes.

One thing’s for sure though – HoloLens will be crunching a massive amount of data. Firstly there’s the processing of data coming in from sensors and cameras in real time and associated environment modelling; and second to this there’s the hologram construction end of business. Altogether processing is handled by a dedicated computer processing unit (CPU), graphics processing unit (GPU) and holographic processing unit (HPU).

Likely it will not be until March that we learn more about actual specifications. It’s then when Microsoft are planning to bring developers into the loop to begin extending evolution of Project HoloLens into new directions, and when HoloLens development kits (may) go on sale.

 

Questions & Limitations

It’s easy to get to wondering about the limits of the technology driving HoloLens. How is HoloLens mapping objects and its environment? And to what extent can a user define what is and isn’t mapped (to create ‘blank’ canvas)? What are the boundaries on projection of objects in depth; or the spatial limits on HoloLens’ sensors?

Given a large enough blank canvas, take the empty volume of a warehouse for instance, it’s conceivable that almost any environment could be simulated. Presumably, limits on scale, or complexity, of a virtual environment will be determined through processing power. Since, the device is slated to be entirely wireless, it’s all begging the question of to what extent one would be able to wander through a virtually-furnished warehouse and how much can be projected…

 

Conclusions

All things considered, it’s immediately clear that Project HoloLens was never about simply about creating holograms. Yes, its engineers have put together technology capable of supporting stunning holograms; but this was perhaps a means to an end. The features of HoloLens appear to come together in a manner that serves to establish a completely new platform for interacting within a digital medium.

The potential here is immense. Blending virtual and digital worlds always seemed the direction for computing – now it seems we have the first working example of just that. Seven years in development? Time well spent Microsoft; well spent indeed.

All things considered, the year ahead is sure to go down as an important one for computing technology: between Project HoloLens, Oculus Rift’s venture into commercial sale and Magic Leap (the project backed by Google with some $592 million, but which remains clouded in mystery), we’re talking about revolutionary tech on the horizon.

 

Resources & References

Microsoft Project HoloLens homepage

Wired article

The Verge article

All image rights – Microsoft