Beyond is a mixed-reality museum experience catered for Gen Z by offering an immersive and engaging journey rooted in multi-sensory interactions, such as voices, eyes, and gestures.


4 Designers (Greg Chen, Wei-Chieh Wang, Youngryun Cho,
Zhuoneng Wang)


5 Weeks, Fall 2021

My role

Research, Concept Ideation, User Interface, Define MR Interactions, Mobile App Prototyping, Video Shooting


After Effect, Cinema 4D, Figma, Final Cut Pro, Illustrator, Photoshop, Premiere, ProtoPie, Spark AR, Unity


The prompt is to design a museum experience for Carnegie Museum of Natural History (CMNH) using emerging technology. We thought it'd be more meaningful to consider their vision while designing for their future.

The vision of CMNH is to become the world’s most relevant history museum by enabling visitors to discover greater stories, including human being a caretaker of the Earth and interconnected relationships between nature and people.

According to the secondary research, to stay relevant, it’s important to engage GenZ (the next generation) in a deeper and creative way and to transform the museum to be more than an information provider, but an experiential and flexible space. Therefore, the problem space we define:

How might we provide engaging learning experiences to GenZ Visitors and enable them to advocate for a sustainable future?


Beyond consists of two parts, “gamified museum experience” to actively engage with Gen Z and “participatory social interactions” to increase CMNH social influence on digital community. We build the concept on mixed reality goggles to ensure the most engaging and immersive experience. First, we introduce a scavenger hunt to create a game-like learning environment, inviting visitors to collect a digital copy of animals or plants they like in the museum. During the process, visitors are prompted to complete tasks with greater stories embedded. The goal is for them to learn while having fun. Furthermore, visitors are able to wear their collections and showcase them as a part of personal style in virtual setting. It also serves as an incentive for museum-goers to visit and revisit the museum. Lastly, visitors are encouraged to share what they learn and their collections as AR filters on social media. To make the sharing interactive and engaging, we design a mobile-based AR filter creation tools to let visitors be creative about their collections. The idea is tailored for the lifestyle of Gen Z. Such social interaction also increases CMHN’s social influence in the digital community.


Visitors are greeted by a sharing wall where past visitors can leave their thoughts with AR filters created with plants or animals collected during their trip. Next to it, it’s a sign directing visitors to start the experience. The wall serves as a space where visitors exchange thoughts, in the meantime, as a prompt for first-time goers to try out Beyond

Visitors get admission wristbands and Hololens2 at the ticketing counter. The admission tag is redesigned as a wristband as it is not only a proof of admission but an anchor point for the glass to display a control panel, leveraging the affordance of a watch

The onboarding gestural tutorial is prompted to users. It also allows users to have a voice command in hands-free scenarios

After wearing the glass, visitors can see other visitors’ collections, encouraging visitors to get their own collection and learn more about these plants and animals

With voice command, the glass can show visitors where to collect desired plants/ animals or any exhibit visitors are interested in

Once a visitor steps inside an activation zone of a diorama, ambiance of habitat in a given diorama would immediately play, immersing visitors in the environment. Visitors then can interact with a diorama with simple gestures, moving a palm as a pointer for hovering, pinching and dragging to grab a plant or an animal from a diorama

By grabbing a plant forward, the visitor could take a closer look at it in 360 degrees, re-vitalizing the dead-dead specimen in an out-of-fashion diorama. They could also learn more facts about the given plant

To collect the plant, visitors would be prompted to a mini-game in the context of the goal of the museum, environmental changes, and connections with humans. After learning through the game, users can collect it to the wristband

Transfer the collection of the day to APP before returning the glass, allowing users to review their trip history and learnings

Visitors can review the latest posts on the sharing wall, their past visits, and learning from the museum, filters, and collection in the APP

Create AR filters using the plants and animals collected in the museum and share their thoughts and learnings on social media. The posts would be synced to sharing wall with the designated hashtag, #CMNH. By doing so, more and more people are engaged, and greater stories are heard.

Design process

Carnegie Museum of Natural History
World’s most relevant history museum
Enable visitors to discover greater stories, including human being a caretaker of the Earth and interconnected relationships between nature and people
Engage visitors in a deeper and creative way
Transform the museum to be more than an information provider, but an experiential and flexible space

Secondary Research

To help the museum stay relevant, we believe it’s important to understand the audiences they attract and the ones they do not because the future of the museum depends on securing the current visitors and attracting potential visitors. Due to confidentiality, we are unable to have visitor demographics of CMNH, we then rely on secondary research to understand which age group visit natural and history museum the most and the least. The data shows:

  1. Museums are successful at attracting audiences between the ages of 35 and 44, 16% higher than the national average
  2. Museums are not connecting with younger audiences (18-24), 23% lower than the national average

Target Audience

We decide to set Gen Z (born in 1996 - 2010) as our target audience as they are the next generation of visitors and donors, however, are least connected with museums. From the research, Gen Z expects museums to be:

  1. Interactive
  2. Experiential
  3. Social shareable, such as instagrammable popups

Field Observations

In our first visit to CMNH, we noticed a few findings:

  1. Need for wayfinding
    Despite a map and signs along the way, it’s still hard to navigate ourselves to the desired exhibit
  2. Week post-museum experience
    The visit ends right after we leave the door of the museum. There’s no way to connect back with museum content and review the learning
  3. Admissions tags are color coded
    Colors represent different day. And we also notice people wear them in their own way. It could be worn as a wristband, a necklace, or put it in their pockets
  4. Lack of channels to share thoughts

explore other museums

Children’s Museum of Pittsburgh

It offers interactive and participatory exhibits, engaging visitors for a longer time. It also uses technology, such as projection and motion detection, to create an immersive environment, allowing visitors to learn while having fun

The Cleveland Museum of Art

Similar to Children’s Museum of Pittsburgh, the museum applies technology to craft a multi-sensory and participatory learning experience. What inspires us the most is the use of Hololens2, it guides users to walk on a curated path to learn and interact with virtual objects from old times

Curator Interview

We speak with Nicole, the curator of CMNH. She shares valuable insights about the role of museums, preferences of visitors, and the vision of the Carnegie Museum of Natural History. It includes:

  1. Wayfinding is always a strong need
  2. Museum to deliver the message of nature and people being interconnected  
  3. Museum to increase the public awareness of environmental changes and people being caretakers of the Earth

Focus Research

Also after the interview with the curator, we spend time studying the museum’s current approach to increase public awareness. Some articles on the website mention several recent renovations are aimed to fulfill the task. Among them, Botany Hall caught our attention, as quoted from the piece,

This refresh emphasizes the Hall of Botany’s relevance in the context of environmental change and the importance of plants in our daily lives

We then decide to focus our research on Botany Hall as it aligns with the latest vision of the museum, giving us a window to see from their perspectives and find design opportunities.

The Botany Hall is known for its great collection of dioramas, and this renovation not only renews dioramas but connects them to the themes. For example, pointing out plants are a source of fiber in human’s daily life helps people realize the interconnectedness of humans and plants, or, by indicating the change of flowering and leading times are in fact affected by human-caused environmental change.

That said, we also find some opportunities for design intervention:

  1. Text-based Interaction
    All the information are communicated through text, requiring visitors’ full and long attention span
  2. Detached Visual Experience
    Dioramas are static and quiet
  3. Unintuitive Information
    Use “Can you find it” game to provide info about plants is not effective. Visitors have to run back and forth between a plaque and a diorama
  4. Missing Context
    Despite the effort of connecting dioramas to bigger stories, visitors are usually drawn to dioramas and don’t spend time reading texts and discovering the story

We summarize our findings from Secondary Research, Target Audience, Field Observations, Other Museum Observations, Curator Interview, and Focus Research:

Define Design Principles

Based on the insights, we define design principles:


We explore and define touchpoints in each phase of user journey. At post museum, we incorporate Social Media as one of the touch points since our target audience is Gen Z, a group that is highly attached to SNS


These are the keywords that guide us during storyboarding,

Provide wayfinding when necessary and remind users of essential steps

With MR glass, users could fully immerse in the habitats of each diorama. For example, users could hear birds chirping in Mount Rainier and wind blowing in Presque Isle

Gamify the bigger stories the museum aims to tell by inviting users to play a mini-game before collecting plants/ animals

Extend the museum experience by encouraging users to review past visits and share thoughts with AR filters of their collection. By sharing, the impact on the digital community would grow and thereby raise public awareness about bigger stories

User Journey of Beyond


Hands on AR/ MR Devices

Since our team is passionate in MR/ VR technology, we decide to design the MR expeirence. We had a chance trying many devices on the market, including Oculus Quest, Hololens 1, Hololens 2, HP Reverb, and Varjo

Admission Ticket Redesign for Accurate Tracking

Although our design is build on Hololens 2, we did not follow the exact paradigm. For example, per MRTK suggestion, palm-up is to trigger menu. We decide to use “raise the wrist with palm-down” for menu. The reason is that raising the wrist with palm-down borrows the affordance of a watch, so it creates less friction for first-time users. It’s especially helpful in a use case like museum where people don’t go very often and need to get familiar with commands right away.
To address the possible increase of false activation caused by palm-down, we thereby add an anchor point on a admission tag for more accurate detection and redesign it as a wristband to reinforce the watch affordance.


We spend much effort simplifying gestural commands because the more gestures we define, the steeper the learning curve is going to be. Considering museum visit is usually short and occasional, visitors should be able to pick up all  commands in a minute. We borrow the affordance of a watch for “Raise the wrist” and “Pinch and collect”.

For hovering, we decide to use palm pointing instead of finger pointing. Based on our secondary research, a finger cannot be tracked by camera very well due to the possibility of being blocked by a hand.

Although some studies suggest the “come on” gesture (left) is the most easily tracked and the most natural hand gesture to summon an object, we think this gesture is not the best choice as it has a negative connotation. We thereby conduct a small scale primary research,  “Pinch and drag” is the most preferred and intuitive in the test.


unity + MRTK + Varjo + leap motion

We prototyped a “pinch and drag” scenario in Unity to demo grabbing a plant from a diorama. We used MRTK Kit as a base and customized it into our design. We then tested in Varjo, a see-through MR headset. It gave us a chance to test the gestural commands and font size. They all worked out well in the prototype.

In the final presentation, we also demo “pinch and drag” use case on a laptop with the help of Leap Motion to track hand gestures.

Spark AR Studio

None of us has experience creating AR filters, so we explore all major AR filter-making software, including Adobe Dimension, Spark AR, and Lens Studio. We go with Spark AR studio in our final production. The tracking systems in the software are inspiring when defining gestures for the project. It also helps us when designing any interactions that involved motion tracking in the future.


We use ProtoPie for high-fidelity mobile APP prototyping.

We take this chance to explore how to prototype with 3D models in the APP, especially for the model to have a 3D rotate.



We have green as primary color and orange as the accent color. For 3D, we go with a low-play, faceted, and fun style, tailored for the taste of GenZ

Hololens UI

In terms of font size, we follow the document by Microsoft to ensure the legibility in MR setting

Wristband UI

Mobile APP UI


next step

The need of Blurred Background

One question was brought up during the critique of the final presentation - do people really need a blurred background in MR world just for better contrast between objects and real-world background? Would it create more problems? Such as safety in the public space, or a detached visual experience. The balance of it would be an interesting topic to explore.

Group Experience

Current BEYOND focuses on individual experience, and we believe extending it for a group of visitors can be meaningful as many people start their museum experience on a field trip at school or on a family trip.


Limitation of gestural commands

We have secondary research on the current technology of tracking gestures in order to define gestural commands for BEYOND. We realize it still has a lot of limitations, for example, if a gesture is performed outside of camera’s view, it will not be tracked correctly. Another example is that because people move their hands a lot both intentionally (for communication and object manipulation) and unintentionally, adding another step to avoid false activation is necessary. Lastly, people’s hands are too useful to be free all the time, so it’s always good to consider hands-free scenarios. I think that’s why META now has a wrist-based neural interface for hands-free and accurate inputs.

Design for MR interactions

I also learn many interactions that make sense for screen-based are not for MR world. For instance, MR devices are not meant for heavy reading for now. Therefore, avoiding text-based interactions is always preferred.

inclusive word choices

One of the comments from the guest judges is to be careful of the word, “climate change”. Some people with a specific political  stand may not be interested in our design at all if we associate it with climate change. Environmental change might be a better term in this case. As an international student, it’s a culture shock and mind-blowing as I was not aware a political preference could be a factor if people would connect to a design or not. It makes me be more aware of the importance and the power of text.

The outcome
The concept
Design process - research
Design process - Synthesis
Design process - development
next step