kioku_banner1.jpg

Kioku

Augmented Reality, Remote Communication Tool

Type              Interaction Design

Duration       6 weeks

Team             Bori Lee, Lisa Li, Chirag Murthy

My Role        Research, UX Concept Development, Interaction Prototyping, Video Making

Tool               Premiere Pro, After Effects, Uniy3D, Cinema4D, Hololens

 

What Is Kioku?

Kioku is an augmented reality communication service that
brings remote families closer through serendipitous moments.

What Can Kioku Do?

CAPTURE A MOMENT FOR LOVED ONES

User can capture a 10-second moment at any time with a simple tap. The moment will be automatically stored in Kioku's cloud system, and a series of tags will be generated to define the context of that moment (i.e., "flower", "daytime", "outdoor").

kioku_capture.gif

EXPERIENCE A MOMENT IN RELEVANT CONTEXT

When another family member passes through a environment that Kioku identifies as contextually relevant with a stored moment based on tag matching result, that moment will appear in the environment as a virtual bubble. By gazing at the bubble and double tap the AR device, users can experience the moment in first-person point of view. 

kioku_experience.gif

REMINISCE THROUGH SAVED MOMENTS  

After viewing the moment on site, user can choose to save that moment into his/her personal library, which can be accessed later by swiping outward on the AR device. When user wanders in the library, unfrequented moments will move closer to the user and start fading if continue to be neglected.

kioku_reminisce.gif

Why We Created Kioku?

OBJECTIVE

CONNECT REMOTE FAMILY

How can we leverage AR technologies to enhance remote family bonding? 
 
How might we synchronize context in a passive an experience-centric way to improve remote communication experience?

SOLUTION

Kioku embraces the physical environment as the new communication interface, allowing remote families experience the mundane daily moments together across time and space.

Since Kioku augments the environment only when it is contextually relevant, it creates a passive and experience-centric communication method that does not require active and focused participation.

 
 

 

01

UNDERSTANDING THE CONTEXT

BACKGROUND

This project is sponsored by Microsoft, who invites us to design a product or service that demonstrates the best qualities of Mixed Reality. The solution needs to bring a positive cultural or social impact.

 

PROBLEM SPACE

The team first defined mixed reality as "combing the virtual and the real worlds". Then we identified the differentiated values of mixed reality as creating "contextual information" and "multi-sensory stimulation".  The team brainstormed design territories that have potentials in using mixed reality, and we eventually choose "communication in a family setting" as our research focus. 


02

EXPLORATORY RESEARCH

RESEARCH METHODS

STAKEHOLDER INTERVIEW

We interviewed a total of 21 people including 10 remote family members. The team became interested in learning more about communication among remote family members as a way of bonding.

OBSERVATION STUDY

We recruited 2 pairs of co-living and remote family members, and observed their ways of communication under a scenario that is common for family living: cooking. 

PRODUCT ANALYSIS

We analyzed 15 current communication tools used by remote families, and mapped them on a 2x2 matrix.

 

RESEARCH SYNTHESIS

kioku_synthesis.jpg

After summarizing our discoveries with sticky notes, we identified 3 main insights: 

 

01 - Remote families are faced with a loss of shared context.

When families are physically living together, they are able to share each other's context instantaneously; when they are apart, they lost such privilege and thus experience communication difficulties with nonaligned or unfamiliar contexts.

 

02 - Mundane everyday moments are often neglected in remote communication.

Even if these moments are shared, most of the time they are verbally described in a fragmented way: information about environments, objects, facial expressions, and other contextual details that that constitute an experience are often lost in the sharing process.

 

03 - Current communication tools cause anxiety and lower the quality of remote communication.

As communication tools constantly notify people to take actions to messages, remote communication is no longer about sharing each other's context, but about ignoring each other's context: the sender injects his/her context into the message, and expect quick responses without thinking about the situation of the receiver; the receiver is forced to pay crazy amount of attention to devices, and may fail to fully engage in a conversation because he/she might be in a completely irrelevant context.

Conversation-Centric = language focused; Experience-Centric = context focused; Active = real-time communication; Passive = leaving some type of messages 

Conversation-Centric = language focused; Experience-Centric = context focused; Active = real-time communication; Passive = leaving some type of messages 

 

Based on these insights, the team determined that our design objective is:

kioku_objective-02.jpg

03

IDEATION & SPEED DATING

IDEATION DIRECTIONS

Based on the design objective, the team generated 3 possible design directions to help us ideating design concepts - synchronize context by:    

  • only initiating communication experience when family members are in similar contexts;
  • creating a new communication environment that invites family members into the same context;
  • providing transparency to each family member's local context.

 

DESIGN IMPLICATIONS

The team brainstormed ideas and conducted speed dating sessions with 10 people. We gathered feedbacks that lead to 3 design implications to guide us moving forward.

IMG_3378 (1).jpg
 
Respect Independence  Several concepts we ideated allow people remotely helping family members in daily chores, which brought concerns of being too dependent or loss of privacy.

Respect Independence

Several concepts we ideated allow people remotely helping family members in daily chores, which brought concerns of being too dependent or loss of privacy.

Carefully Treat the Boundary   People are sensitive about the boundary between real and virtual worlds. While remote members yearn for seeing each other, they don't want to pretend that the distance does not exist. 

Carefully Treat the Boundary 

People are sensitive about the boundary between real and virtual worlds. While remote members yearn for seeing each other, they don't want to pretend that the distance does not exist. 

Support Current Practices  People communicate to get to know each other's context, and if all contextual info are constantly shared by a new system, people might lost the motive to initiate remote conversations.

Support Current Practices

People communicate to get to know each other's context, and if all contextual info are constantly shared by a new system, people might lost the motive to initiate remote conversations.

 

 

CONCEPT FINALIZATION

The team therefore proposed to design a passive and experience-oriented communication tool that challenges people to rethink what it means to communicate:

How does it feel to know your dear one's life without the fatigue of waiting for responses and replying instantly?
How does it feel to encounter serendipitous moments and experience your dear one's life in similar scenarios and settings? 

04

CONCEPT DEVELOPMENT

Our final concept, which is named Kioku (Japanese word for memory), is an augmented reality communication service that allows remote families naturally share everyday little moments in similar contexts despite time and location differences.  

USER JOURNEY & INTERACTIONS

 

INTERACTION DETAILS

After defining the basic user flow for the system, the team went on to detail out specific interactions within each component based on different scenarios. 

Screen-Shot-2017-09-12-at-10.35.27-PM-1.png
SAFETY LOCK - If user is in dangerous situations such as driving or is viewing fictional content such as watching a movie, Kioku will identify these contexts and not display any moment notification.

SAFETY LOCK - If user is in dangerous situations such as driving or is viewing fictional content such as watching a movie, Kioku will identify these contexts and not display any moment notification.

COLLECTED MOMENTS - If user captures a series of consecutive moments in a short period of time, Kioku will display them as one moment cluster, and attach them to relevant objects.

COLLECTED MOMENTS - If user captures a series of consecutive moments in a short period of time, Kioku will display them as one moment cluster, and attach them to relevant objects.

MULTIPLE MOMENTS - If user captures multiple moments at the same location in different days, Kioku will display the older one first, then newer ones once it is consumed. If multiple members capture  at the same location, Kioku will differentiate the capturers by color.

MULTIPLE MOMENTS - If user captures multiple moments at the same location in different days, Kioku will display the older one first, then newer ones once it is consumed. If multiple members capture  at the same location, Kioku will differentiate the capturers by color.

 

SYSTEM DESIGN

Kioku embraces the physical environment as the communication platform and present augmented information in a context that relates to both the sender and the receiver. 

System Map

System Map

 

BACKEND LOGIC

Kioku leverages current machine learning technologies for context recognition. After a family member captures a moment, Kioku will analyze and generate a set of categorical tags that define the context of that moment. When another family member uses Kioku elsewhere, the system will constantly scan his/her physical surroundings and detect if any stored tag combination matches with the current context. If there is a match, the related moment will be retrieved from the cloud and ready to be experienced.

Backend Logic Diagram

Backend Logic Diagram


05

PROTOTYPING

NOTIFIER FORM

We made some quick image collages to test which form works the best as a moment notifier, and we chose the crystal ball form in the end. Here are some of our key findings:

  • Organic forms are more approachable but less obvious compared to perfect geometric forms;
  • Solid shapes with no transparency would block user's views;
  • It's hard to define the boundary for contextual notifiers (highlight a random object in the environment).  
 

INDOOR SPACE

2D GRAPHIC NOTIFIER

3D CONTAINER NOTIFIER

CONTEXTURAL NOTIFIER

OUTDOOR SPACE

 

NOTIFIER PLACEMENT & INTERACTIONS

The team quickly made some low fidelity prototypes to test the user experience with the notifier and the moment itself, and we found the following insights:

  • The most comfortable setting for moment viewing screen is a curve plane that covers 110° horizontally, and 50° up/70° down vertically of the visual field;
  • The sphere notifier has the most natural yet noticeable presence if its diameter is between 20-25cm within field of view;
  • Using sound as the moment notifier instead of visuals will not work if the user is in a noisy environment;
  • Using long-gaze gesture to open the moment will not work for crowded environment. Instant gaze to select and double tap to open is more feasible. 
kioku_prototype.gif

06

NEXT STEPS

To continue with this project, we'd like to:

  • including on-boarding experience for the service;
  • Design interfaces that allow user to manage moments in the personal library;
  • Prototype the experience with Unity + AR SDK.