Time changes everything — fashion, music, trends, landscapes. Take Chicago for instance. Known for its blues, world-famous hot dogs and architecture, Chicago has evolved dramatically over the years.

Yet, almost any night of the week you can head to Kingston Mines and experience those historic blues. Likewise, you can walk a few blocks and find yourself a Chicago Dog about anywhere in the city. On the contrary, it’s often difficult to visualize the changes in Chicago's architecture over time. Historic buildings are renovated, new towers alter the skyline and bricked roads are covered up in favor of pavement. That's precisely why we at Perficient Digital Labs created ChicagoAR.

Old Meets New

ChicagoAR is an augmented reality app that allows anyone with a smartphone to view Chicago's past. Like opening up a wormhole, ChicagoAR utilizes augmented reality and data anchors (the white signs seen in the video above) to superimpose historical photography upon its modern landscape in life-size proportion.

Each AR experience is location specific and showcases a different era in Chicago's history. Once augmented reality is superimposed, the user can move freely, view the scenery from any angle, or even approach the life-sized augmented projection to inspect it up close (just be careful of cars).

A Look Behind the Curtain

For those interested in the technology behind ChicagoAR, check out our other articles where we breakdown our design/development approach and detail the tech stack.

ARKit Image Recognition: Testing Bounds and Capabilities

CTA-AR: Enhancing Chicago public transit with Augmented reality

Conner Hasbrouck is an Engineer & Designer at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

ChicagoAR: Old Times & New Tech was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Want to skip to the video? It’s here.

It seemed simple at first. The idea was to make an Electroluminescent Light-based signaling system. Some switches, some lights, some wire and we’d be done, right?

As usual, we challenged ourselves to produce a practical wearable product. What can we produce that’s new, interesting, and useful? A common scenario: Short time spans and high demands.

Even for internal exercises we need to flesh out the concept. We went through the process that we go through whenever we’re presented with a new challenge: Who is it for? How will it be used? What is it supposed to do? What problem will it solve?

Asking the right questions: How UX Design Discovery Works

For software, there are many tools and techniques for user testing and experience validation. It’s trivial to collect data points on a user’s behavior, correlate that with other sources, and build a pretty report on the recommended course of action.

It’s not as simple for hardware. Each interaction from every user will be different than another to some degree. It’s the nature of the physical world. Additionally, in the prototyping stage, it’s difficult to iterate as quickly as you can with software. It’s much easier to swap out some lines of code than a knob or switch. The decisions have to be made, primarily, off of qualitative data.

Let’s walk through our process, from concept to prototype for this product…

🧠 Round One: Concepts

Our original idea was to make a signaling system for bicyclists. That meant the controller needed to be lightweight, easy to operate and unobtrusive. The indicators needed to be simple, clear, and useful to both the rider and the people they share the road with. The whole system needs to be portable and adaptable to any bike of any size.

An insight: As silly as it might look or feel, “Bodystorming” is a great way to prototype physical interactions. Microsoft has some great insight on this process. Obviously, it’s imperative that you keep the user top-of-mind throughout the entire design process.

Our initial thoughts for the controller revolved around attaching something to the brake handles. But we had concerns about the portability and obtrusiveness of that installation. What about a hand mounted system? What if the controls were your fingers?

Always a fan of the Nintendo Power Glove, it served as our original inspiration. As something more practical for a bicyclist, we chose a generic ‘cycling glove’ off of Amazon and went with it.

For the indicators, we wanted to use Electroluminescent Paint to create something like what you see in ‘festival masks’. However, sourcing the ink in time and at a reasonable cost was difficult, so we went with the old standby EL tape.

We settled on a backpack as a mechanism to mount the tape to, and as a container for the various electronics needed to run the system. It seemed feasible to us, that were this productionized, it could be reduced to a size that could be placed in/on a real backpack.

To control the EL elements, we needed to use a relay board. This allows us to isolate the low-voltage DC that controlled the arduino from the High Voltage AC that powered the lights.

⚙️ Round Two: Prototyping the prototype

The glove fit well and offered many different methods to attach complementary components to control the lights. We started with magnetic switches, but they were too bulky. Alternative ideas revolved around ditching the glove and switching to some type of inertia tracking device or vibration sensors connected to various places on the bicycle. However, nothing was as simple and direct as manual manipulation.

We settled on buttons under the fingers. There’s several different types of switches that we could have used, and even more models of each of the different types. But through testing we found small, simple momentary push-button switches to be the easiest and most reliable option.

A discovery: Grab a pen in the same way you would grab a bike’s handlebar. You can feel that the majority of the pressure isn’t under your fingertips as you might assume at first. It’s actually under your finger's first and second set of joints. We needed to mount the buttons up towards the middle of the glove’s fingers to ensure maximum pressure and control on those buttons. Additionally, we mounted the buttons on the inside of the gloves to allow for a better feel of when the buttons were being pressed.

Well, that’s it. Press some buttons, turn on some lights… cool, I guess? Not yet, we needed more…

🔥 The Pivot: What’s hot in the streets?

It was around this time that Chicago launched their e-scooter pilot. And with that came all the controversy and intrigue about the new two-wheeled menaces cluttering our sidewalks and bike paths. The scooters themselves are IoT devices. Tracked, controlled, and activated through various combinations of bluetooth and cellular communication.

A discovery: Bicyclists already have an established hand signal system to indicate turning, stopping, etc… Scooters, on the other hand, must be controlled by two hands. The slightest accidental movement at speed will send you tumbling down into the asphalt. Our single-handed indicator system could solve two problems at once.

For this to be timely, we pivoted to envisioning this as a scooter indicator system. Additionally, we needed to spice it up with some Bluetooth Low Energy communication.

🥶 Feeling a little blue

Wiring directly from the glove to the backpack worked, but it wasn’t very cool. Luckily, we had some spare BLE-enabled devices laying around and decided to put them to use…

After laying out the prototype for the BLE interface, we quickly realized a key opportunity to enhance the physical UX for the rider. Instead of pressing the buttons to turn on the signals, what if the rider triggered the signals by lifting a finger? This would automatically turn on the brake light whenever pressure was released from the throttle.

To test this hypothesis, we included a switch on the protoboard which changed the switch values from NO (normally open) to NC (normally closed) and vice-versa.

A discovery: In code, you can implement an A/B test with a simple if statement. However, in hardware it’s a little more difficult. Consider ways to test implementations with spare input or output mechanisms that affect the behavior of the device. Obviously, you don’t want to introduce any variables that could affect the test.

A second discovery was that if the rider is riding with the indicators on their back, it’s hard to validate if they’re working correctly. A new technique was introduced to our repertoire: Audio debugging. We placed a piezo buzzer on the protoboard which would change it’s sound depending on which buttons were being pressed, if any.

🛴💨 Let’s Roll!

Dan Thompson is Director of Emerging Experiences at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

In an experiment, most links on this post are Amazon Associate links. As an Amazon Associate I earn from qualifying purchases from those paid links.

It fits like a glove: The UX of hardware prototyping was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you've watched any tech conferences in the last few years, one thing is very clear: companies and their communities are excited about augmented/mixed reality.

Apple has dedicated time in each of its last three WWDC conferences to talk about its ARkit technology. Microsoft is leading the wearable charge and soon releasing the second iteration of its Hololens headset. Startups like Magic Leap have penetrated the AR market and tech moguls like Facebook are following suit and releasing their own tools like Spark AR.

Despite all this attention and talk of innovation, something about AR technologies just hasn’t clicked yet. Yes, we have Snapchat filters and apps like Pokemon Go and IKEA Place. But where is the futuristic society where AR technology is helping us navigate to our destination or shop at the grocery store? Why is AR not assisting us with our work and guiding us through the tasks of everyday life?

Simply put, the medium of traditional AR experiences are still lacking.

The Problems

The Hololens was a groundbreaking Mixed Reality headset. Unfortunately, it’s also a large and clunky piece of hardware. It has a limited viewing area and an expensive price tag. It’s not a mainstream device, and Microsoft is acknowledging that by positioning the Hololens 2 even further into the professional/corporate market.

ARKit and ARcore have made significant capability jumps since their announcement, but they still limit your AR experiences to a phone/tablet. Accessible? Yes. But limited in input variance and computing power.

The magic of the Hololens and Magic Leap is that you are surrounded by Mixed Reality. Mobile device screens limit the experience to small windows into the Augmented Reality world.

So where does that leave us? Well, we could just wait another ten years for the tech to advance. Or, we could create Augmented/Mixed Reality experiences in another way…

AR, VR, & MR: A brief, opinionated, overview of the major platforms

A Different Take on Augmented Reality

This isn’t CGI or something on a screen. It’s a real shoe, changing color in real-time, at the press of a projected button. It’s augmented/mixed reality in an entirely different format.

Here’s what that shoe looks like under normal lighting conditions.

Another installation we put together utilizes the geometry of an exercise ball to create a unique 3d data visualization:

So how is all of this possible? With projection mapping and computer vision.

From Screen to Projection

In this context, projection mapping is the process of turning a three-dimensional object into a display surface for video content. To accomplish this, you need some way to analyze the bounds and depth of an object. With this information, you can project video content solely to the desired surface while also accounting for depth variance. After all, we are working with 3D objects. Not a flat, 2D, forward-facing plane. If unaccounted for, any video projection would become skewed when projected at an object with variant depth.

Until recently this has not been an easy task. Its required advanced technical knowledge and some pretty expensive equipment. However, affordable tools are beginning to emerge in the market to simplify this process.

One such tool that we utilize is Lightform. This piece of hardware streamlines the depth analyzation process and is a rather affordable solution. Another beautiful thing about this tool is that is integrated with most market projectors (with HDMI input) and ultimately removes the needs for specialized video mapping projection equipment. Here’s what our setup looked like:

How is this Better?

Compared to AR experiences on a headset or phone, projection mapping exists further along the spectrum of mixed reality. The illusion it creates exists in the real world, rather than a screen. This opens up viewership to anyone present, rather than the sole user of a headset or phone. Now everyone can be included!

Perhaps even more important, this flow opens up a broader spectrum of input interactions that are inaccessible to traditional AR experiences. The shoe’s input menu is also projected, but ‘touching’ is recognized through computer vision (In this case OpenCV). We can actively track those projected buttons and watch for user interaction. Upon detection of a users touch, we can trigger color-changing events in our shoe configurator.

Using this flow, we can create “projected inputs” that exist on any surface, in any medium, shape or color. We could trigger them with touch, human movement, facial expression, whatever you want — All through the combined capabilities of projection mapping and computer vision.

The Road Forward

Projection mapping feels like an untapped gold mine, lying in wait for the creative community to take advantage of. While it does come with its own set of unique obstacles it pushes AR/MR experiences in a direction previously out of bounds. It’s not a complete solution, but it begins to answer some of the biggest questions preventing Augmented Reality from breaching the mainstream market.

Furthermore, in the evolution of the AR/MR medium, it’s important that we continue to experiment and think about mixed reality in creative and unique ways.

Conner Hasbrouck is an Engineer at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

Breaking Augmented Reality out of the Screen was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Virtual Reality is inherently a solitary experience. You shut out the real-world once you put on your headset. Your vision is filled with an alternate reality, where wands and controllers approximate your real hands. You might interact with avatars of others in your virtual world, but they could be continents away.

As part of an internal design and development challenge, we set out to change that experience. Perficient Digital Labs’ designers and developers prototyped three different scenarios to explore ways that the real world could be brought into the virtual world. From these exercises, both groups gained knowledge of concepts that apply to VR: software development, interaction design, IoT concepts, and hardware prototyping. We call the results ContactVR.

What if you could share a VR experience with someone without actually being in VR yourself?

We started with a simple question, “What if you could share a VR experience with someone without actually being in VR yourself?”. How could we affect someone’s Virtual Reality from outside the headset? What real-world interactions can we harness to affect a virtual world?

ContactVR fosters real-time connections between a VR user, custom controllers, and real world participants. It also allows for multi-person experiences. This allows for more social interaction outside of the headset.

The Scenarios

There are three scenarios that we explored:

• Multiple real-world users can interact with a Virtual user through their mobile devices. (Many-to-one)
• A Teacher & Classroom scenario where a teacher can control the student’s immersive experience. (One-to-many)
• A training simulator where a user interacts with a custom controller to simulate control of a virtual machine. (Simulation)

Space Assault

In this scenario, a user plays the virtual role of an astronaut surrounded by alien creatures. Each alien is actually a person in the real world who can blast the astronaut using a web-based mobile interface on their phone. The astronaut’s only line of defense is a shield that they have to block the alien’s beam.

This game utilizes web sockets to communicate between each real world user’s phone and the virtual user’s headset. The bi-directional communication is near instantaneous and shared among all of the users (real and virtual). Scores and status are updated in real time on all devices. In the virtual user’s headset, the Unity application utilizes spatial audio to alert the player of the real-world user’s blast.

Solar system explorer

This scenario supposes a teacher walking through a lesson with students. At the appropriate time, the students enter the virtual world and are given more in-depth instruction in virtual reality. As the teacher describes specific planets, they select the planet on their phone. This action causes all of the student’s views to focus on the teacher-specified planet.

This guided educational experience offers a new and engaging way for students to learn together, at the direction of the teacher. While they can still explore their virtual reality individually, the teacher ultimately controls their progress and focus throughout the lesson.

ExcavatVR

To explore ways that a user in virtual reality could interact with the real world on an individual basis, we decided to explore how a custom physical controller could influence an object in a virtual world. Driving and controlling an excavator from virtual reality easily won the most support.

To accomplish this, we wired a series of buttons to an Arduino and connected that to a Raspberry Pi. With some custom software, we were able to send button presses through the cloud to the headset which then controls the excavator. While the controls don’t reflect an actual excavator’s controls, this concept could be tailored to training experiences, allowing a user to learn how to control a complex or dangerous piece of equipment in VR before venturing out into the real world.

Entertainment, education, and industry are only a few areas in which this concept and technology can be applied. Bringing the real world into the virtual world opens a new way to think about Mixed Reality. The possibilities are endless for shared experiences.

If these concepts are interesting to you, and you’d like to learn more about what Perficient Digital Labs can do for you, let us know.

Dan Thompson is Director of Emerging Experiences at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

Beyond the Headset: How real-world interfaces can augment Virtual Reality experiences was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the most common topics we’ve heard clients ask about over the past few years is the concept of smart spaces — how we can incorporate devices, sensors, people and places into a system that helps us learn about our space and in turn, use our space better. Our most recent foray into this topic area explores what could be done to help folks learn about their space in a low-friction way. Introducing Odyssey — the automated tour guide.

As you move through a space, the Odyssey Progressive Web Application (PWA) provides location-aware content, powered by a connected Bluetooth beacon and receivers located throughout the space. From a user’s perspective, the experience is simple, the app does the heavy lifting. Picture walking through an art museum, and as you walk up to a sculpture, the app seamlessly transitions to show a description of the piece, an artist bio, and a slideshow of initial sketches for the piece. That’s Odyssey in a nutshell.

The components

Behind the scenes is a more complex story — a connected system made up of three parts: a Bluetooth proximity beacon, a Raspberry Pi receiver, and a progressive web app (PWA).

• Bluetooth proximity beacon — The beacon is used to track the user’s location relative to the receivers in the space. For this project, we used a Gimbal Series 10 Beacon — a small device that could be dropped in a pocket or attached to a lanyard.
• Raspberry Pi receiver — The receiver recognizes the beacon’s location and communicates with the PWA to update. We selected a Raspberry Pi for this implementation, but our solution could easily be implemented on any IoT device with Bluetooth capability.
• Progressive web app (PWA) — The PWA houses the map and location content. A PWA is just like any other web application, but it offers features that are only available in native apps. We used the power of PWA to enhance Odyssey with features like bookmark creation and offline functionality. This is a low-friction option because there is no app to download, the user can just visit a website. Read more about progressive web apps here.

To tie all the systems together, we utilized Microsoft Azure IoT Hub. IoT Hub allows us to process real-time device telemetry, implement custom visualization, manage devices, and collect vast amounts of data upon which we can perform analysis and actions upon.

How it works

These three components come together to form the Odyssey system. To demonstrate how it works, we’ll use the office example seen in the video above.

1. A new employee is provided a unique beacon.
2. The employee scans the QR code on the beacon to pair it with the PWA. On scan, a link to the PWA appears. This is a unique link that tells the the PWA to associate with this specific beacon.
3. The employee carries the beacon with them as they tour the space.
   As the employee enters a new room in the space, the receiver in that room recognizes the beacon’s location and tells the PWA to transition to that room’s content screen.
4. The employee can look at photos and videos and read content relevant to the room they are currently occupying. This multimedia content is managed through a CMS like Contentful.
5. When the employee exits the room, the PWA transitions back to the default office map.

We also added additional features to the PWA: an onboarding tutorial that teaches the user how it works, and a freely explorable 3D map for users that want to take a more manual approach to learning about the space.

Where it works

Though we designed and built the app with an office orientation in mind, we think Odyssey can be useful in many different scenarios. One key benefit is that it’s great for indoor tracking where GPS doesn’t work very well. That makes it ideal for use cases like:

• Residential building tours
• Museum tours
• Campus orientation
• Hospital wayfinding
• Trade show/convention wayfinding
• Factory/warehouse training

There are tons of possibilities with Odyssey, we’ve only scratched the surface.

Do you have ideas about how Odyssey could be used? Want to learn more or schedule a demo? We want to hear from you! Email us at hello@perficientdigitallabs.com and we’ll get the conversation started.

Further Reading

• Enhancing Chicago’s public transit with Augmented Reality: Sign Detection and Route Visualization
• Data Spaces
• Beacon tracking with Node.js and Raspberry Pi

Tyson Damman is Creative Director at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

A new way to experience space was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

How UX Design Discovery Works

“What dat?” asked my two year old niece twenty or so times as she pointed at almost every item in the room. I happily answered every inquiry to the best of my ability and watch the little gears turn in her head. Imagine starting from scratch, trying to understand the world one item at a time with a single question — the mixture of frustration and excitement is clearly visible on her face.

I often feel a bit like my niece as we begin a new project, trying to understand a world that I’m not yet familiar with. Luckily this isn’t my first rodeo, I come prepared with an arsenal of focused questions to get to the root of the problem we are setting out to solve. At Perficient Digital Labs, we employ a methodology designed to break down even the most complex of problems in the most complex of industries into simple, surmountable elements. Let’s talk about how to perform one of the most fundamental parts of the design process: how to ask questions.

Discovery

Before beginning a single sketch, choosing a typeface, or writing a single word of code, we launch every project with a phase called Discovery. Discovery can last from a day to a few weeks and can take many forms, but essentially, it is the method of understanding a client’s business, its customers and the specific problems at hand.

Why do discovery?

As consultants, we’ve worked in nearly every industry you can imagine from healthcare to finance to journalism. Even with an extensive catalog of experience, it’s impossible to know all the ins and outs of a client’s business off the bat. One of the great privileges of being a consultant is the chance to learn about how a particular business works. You don’t know what you don’t know, and the only way you can know is by speaking with the experts on the subject, client stakeholders.

Sometimes, the jargon and veils of industry complexity can be roadblocks to a successful project. While it’s important to understand the language of a client’s business, you can easily break through any roadblocks by resetting and viewing the project through the lens of the customer. User experience design is inherently user-focused, heck it’s right there in the name. Every activity in the design process must consider the user first and foremost. Discovery is your chance to meet your client’s customers and get to know as much as possible about them.

How it works

Discovery isn’t cookie-cutter—it could be anything from a multi-day workshop to a 1-hour phone call. Each project’s discovery phase should be tailored based on several factors:

• client — is this a new client or someone you’ve worked with before?
• size of project — is this a multi-phase project or a short sprint?
• complexity — how complex is the problem? is it a brand new product or a redesign?

Once a format has been established, the next step is gathering the right people. A team generally consists of 3-4 members. A director runs the sessions and guides conversation. A producer tracks project logistics like scope and timeline and captures detailed notes. A designer and/or engineer also takes detailed notes and aids in running the sessions. Client teams work best when project leaders and subject matter experts join forces to give a full picture of the task at hand. Project leaders answer big picture questions and subject matter experts answer the nitty-gritty detailed questions.

A discovery session can be made up of any number of these activities:

• Q & A — A round-table question and answer session aimed at gathering background and requirements.
• Demo — A client subject matter expert (SME) walks through the product to demonstrate key functionality and pain-points. During this walkthrough, the design team asks follow-up questions and captures requirements that arise.
• Observation — The design team shadows a SME, or more ideally, a customer as they go about their duties and interact with the product. Questions are asked to identify use cases, priorities, and pain-points.
• Concept sketching — Whether in a workshop setting with clients or back at the office, the design team guides a rapid sketching exercise that focuses on key requirements. The goal is to generate a ’silhouette’ of what the product could be before diving into detailed wireframing sprints.
• Prioritization — Clients participate in a requirement ranking exercise to focus in on the most pressing issues from a business and customer perspective.
• Technical discovery — Engineers meet with relevant technical SMEs to determine systems, data components, limitations etc. Note: In this article, I will not dig into technical discovery, instead focusing primarily on user experience design.

At the root of every activity is the basic methodology of asking the right questions. You cannot expect your clients or their customers to lay everything out on the table for you. Even the most organized clients cannot possibly present a 100% crystal-clear picture of what they need. It is our role as designers to facilitate and guide this process, focus where necessary, and avoid unproductive traps. This doesn’t happen magically, one must come prepared.

How to ask the right questions

Every industry and business is unique, but when it comes to user experience there are similarities all over the map. We don’t start from scratch every time we start a new project because there are core issues every project shares. With a common set of questions that aid in establishing goals, understanding users and defining product components, you can turn complex problems into elegant digital solutions.

Setting Goals

The most fundamental thing to understand is what you are building and why you are building it. You may have a pretty good grasp on the answers to these two questions before starting the project, but it’s important that all parties get on the same trajectory before embarking. For example, a project leader may understand the reason why a project is being undertaken from a business perspective (to gain users, to expand feature set, to cut revenue losses) but a technical SME may understand how the codebase needs to change in order to accommodate a shift in user experience. For this reason, ask these fundamental questions with all stakeholders in the room. It’s extremely important that all voices are heard and expectations are set early to avoid issues down the road.

Some of the common goal-oriented questions to ask are:

• Why are we doing this project? Pretty basic right? But you will often get differing answers from different stakeholders and it’s crucial these answers are heard.
• What are the biggest obstacles we need to overcome? Often the reason you are being hired for a project is to solve issues that could not be solved internally. You need to locate these pitfalls and find ways to avoid or bridge them.
• What does success look like? This question does more to define project goals than any other in this list. Clients usually know what they want, they just need your help to get there.
• How could we fail? The counterpart to the success question, a question about failure helps understand where to focus attention to come away with a win.
• What is the story we want to tell about this project 3 months/6 months/1 year down the road? Ask clients to visualize a press release or an article after the project has launched — what is the headline? what are the key takeaways from the article? what will people be talking about long after your product is introduced to the world?

These questions help establish goal statements that all parties can agree on. Every requirement, concept, or design component that arises should align with these statements. You should be able to point to these statements at the very end of a project and confirm that what was built has fulfilled the project goals.

Understanding users

All good design is empathetic. It is our charge, as user experience designers, to center everything we make on the people that use it day in and day out. The term “user” is a bit robotic, but for lack of a better option, I’ll be using it to represent these folks. A user-centered design process doesn’t always require creating detailed persona profiles, down to what variety of apples they like (though that might be very useful if you’re creating an app for apple critics). But it does require a detailed understanding of how they do their job, what frustrates and excites them, and how they might ultimately use the product you’re building. The first activity is to list every type of person that might use this product, no matter their level of involvement. In order to focus conversation, prioritize these user types. This has long-term importance as well, especially if you define your sprint schedules based on user types. For example, a team may focus more design cycles on the everyday user (90% of user base) than they would on the admin user (10% of user base). The next step is building a profile for each user type.

To learn about the users and their behavior, ask questions like:

• What tasks does this tool help the user accomplish? How does the user accomplish these tasks today? This is how to determine areas of functionality to build for each user type. You should also understand current user behavior to inform new functionality. Maybe the user employs an excel sheet to track some data today. You will want to know this in order to assess whether it’s functionality that can be incorporated into the new product.
• How often will the user use this product? What actions does the user take on a minute/hourly/daily/weekly/monthly/yearly basis? Time-based questions help define priority as well as understand behavior. If a user is spending their entire workday in the product, there are ergonomic considerations that don’t exist for a user that checks the tool once a month. Frequency questions help determine priority within the product — things that are done more frequently generally require more attention.
• What does the user ask for help on most often? By identifying user pain points, you can further refine your focus on solutions that will have the biggest impact.
• Has the user created any work-arounds to remedy current user experience problems? Often, folks have created clever work-arounds when a product isn’t working for them. Sometimes these solutions are analog, sometimes digital. Identifying them helps determine desired functionality.
• What does this user need to be able to communicate to other users? Is this a solo or collaborative product? What functionalities will help users communicate?
• What are the most typical pathways this user might take through the product? This question acts as a segue between user questions and functionality questions. You will probably spend the most time discussing this question in your discovery sessions. It requires a bit of foresight in order to visualize potential user scenarios. Focus your functionality questions on the user flows outlined in the client’s answer.

At this point, you should have a solid understanding of the key users, why they use the product, and how they might use it. Next, you’ll examine potential functionality for each user type.

Features and Functionality

As a result of the user-based questions, you have identified at least a few key functional components or features for each user type. The team then dives deep into each of these components. This is where sketching exercises may be incorporated to get some early functional ideas on paper, but at the root, you’re still asking questions. You are trying to understand the same things you need to understand about the overarching project, but on the feature level — What is the purpose? Who uses it? What is the flow?

Focusing on one key functional component at a time, ask questions like:

• What is the purpose or goal of this feature? A seemingly basic question that can sometimes determine whether or not a feature is worth building. The purpose of any feature should also align with the goals of the product.
• Which user type(s) will use this feature? Is authorization required? Who uses a feature gives you an indication of how important it is and where it should live in the product. It also may determine how complex it can be, based on the user type’s skill level. If authorization is required, there may be unique states of the component to be designed as well as a method for assigning permissions.
• At what point in a user’s process would this feature be most useful? You can assess whether a component has permanence or is a temporary state.
• How does this feature change over time? Much like authorization, this question helps determine the states required and how the system changes based on each user’s process.
• What are the available options in this feature? How can it be customized? In addition to determining component states, this question may dictate whether you will need to design a method for customization.
• Demonstrate a simple and complex usage example of this feature. This may require some imagination, but it helps the team visualize the range of experience that must be accounted for.

Data usually plays a key role in the products we design. To get a solid understanding of how data will flow in and out of the product, ask:

• How will a user use this data? Are they scanning it quickly to find anomalies? Are they correcting fields? Are they inputing data in forms? There are many answers to this question and all of them change user experience significantly.
• What are the minimum / maximum / average number of items? A component containing 5 data points would be designed quite differently than a component with 1000 data points. This question also helps determine the variability of data.
• What are the options for filtering/sorting? Determine the most useful tools for slicing and dicing the data.
• What artifacts/output would be useful to a user? A user might need to share a PDF report with their boss. In this case, you’ll want to provide a method for easily sharing the data.

It’s rare that every question for every feature is answered during the formal discover sessions, but the primary goal is to build a basic scaffolding that features can be built upon. The kick-off meeting of each design sprint is another great opportunity to dive further into functionality that was not defined during the discovery sessions.

Moving forward

I’m going to avoid the nitty gritty details of the post-discovery design process for this post, but it is crucial to understand that the answers gathered in discovery become the guideposts for all work that follows. Every overarching system you create, every component you design, and every button you detail is a result of asking the right questions. I feel strongly that this process is the only way to guarantee a tailored, innovative user experience, designed in an efficient, intelligent way. Sure, you could start a project based on an email containing some basic requirements, and it might look pretty good. But chances are, that resulting product will lack true understanding of a client’s business and customers and be nothing more than a pretty piece of wallpaper.

The user experience design process requires that clients are active and engaged in the work being done. The most successful projects we’ve experienced are truly collaborative starting with an energetic, mind-melding discovery phase and carrying that spirit through design and engineering sprints. Design consultants bring the experience of working with dozens of clients on hundreds of projects, each one informing the next. Clients bring their wealth of industry, customer, and process knowledge. All of this knowledge combined can create beautiful, innovative things — but it all starts with the right questions.

If you’d like to get to know our process a little better or have an idea you’d like to make a reality, please say hello at hello@perficientdigitallabs.com

Tyson Damman is Creative Director at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

Asking the right questions: How UX Design Discovery Works was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ve discussed the great aspects of USDZ and iOS Quicklook, now I’ll review some real-world issues we’ve run into, and some concerns that we have with the format.

A single file containing a packaged 3D scene. Easy to share. Great looking visuals. Lightweight file-size. Built for augmented reality.

Sounds too good to be true, right? Well, there are a few issues preventing USDZ from taking over right now. Thankfully we think these are just temporary speed bumps and not critical flaws.

First, is the typical walled-garden of Apple approach of using an alternate file-format than other major platforms. Established formats such as Collada (.dae) could likely be used for the same utility. More glaringly, however, is the refusal to adopt glTF, which is quickly becoming the new standard for sharable 3D assets.

Next, is the tooling. usdz_converter works well, but browsing the usdz-interest google group reveals that the format is still in its infancy and there are few tools and plugins that are fully baked. Certainly, new tools like Adobe Aero, and the recently acquired Allegorithmic Substance Painter will continue to emerge with USDZ support, but right now it’s not a universal format for many platforms.

My Experience

When using USDZ as a 3D file format in an iOS ARKit application, I experienced an apparent lack of control with those files. Many of the editable properties and instance methods available for alternative formats (SCN, OBJ) have no effect on USDZ files. Here are a few of the issue I ran into while using a USDZ in an iOS application:

• Controlling the replay/loop of animations. USDZ files loop by default with no apparent way to change/control that behavior. It’s all or nothing.
• Stopping/Pausing the animation at specific animation timestamps. The callouts for animation start/end don't trigger.
• Changing the scale of the USDZ file. Whatever size you import is what you have to work with. This appears to be based on the root node of the geometry.
• USDZ / Quicklook is well suited for previewing objects. Attempting to create large, complicated scenes will not work well for you. There are better technologies and methods available for this functionality.
• Starting the animation from the beginning. USDZ files appear to cache themselves. If you reinstate a USDZ model or restart the app, the animation will start from the frame you left it on, rather than its first frame.
• On your desktop, the Xcode 3D tools do not allow you to edit properties of a USDZ file. Nearly every click asks you to convert it to a SceneKit (SCN) file.
• Additionally, the Xcode 3D tools do not accurately preview the PBR textures of a USDZ file. The only accurate way to preview USDZ files is on your iPhone.

The Future

Aside from of Quicklook, it’s apparent that support for USDZ is not fully baked yet. Right now the primary purpose of USDZ files is to be a lightweight, shareable model for preview.

At this stage, USDZ files don't feel supported enough for advanced/interactive ARKit applications. Yet, I assume that Apple has greater ambitions for USDZ that include full-fledged support in Xcode and ARKit. The creation flow will be simpler, the handoff is seamless, the models look fantastic and they are memory conservative. This is definitely a topic to to pay attention to.

Conner Hasbrouck is an Engineer at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

Adventures in 3D — Pitfalls and problems with USDZ and iOS Quicklook was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

As a follow-up to my last post, I wanted to identify a couple processes and a tool that I utilized to hasten the preview and editing process of USDZ files. If you have any additional tips, or feedback, feel free to leave a comment at the end of the post.

iCloud + symlink

Currently, the only effective way to preview a USDZ file is to view it on your iOS device. That transfer process can become cumbersome and result in many iterative files if you’re using AirDrop or sending the files to yourself through a messaging app. Instead, use iCloud to sync a folder (such as ~/Desktop), and create a new folder that will contain your USDZ file.

Then, create a symbolic link in your working folder to that iCloud folder using this command:

ln -s ~/Desktop/usdz usdz

That will create a folder ./usdz that points to a synced folder on your Desktop, which is synced to iCloud by default. Every time you write out a new .usdz file to that folder, it will be synced to iCloud and will be automatically available for you to view in the Files app. I found this to be much faster than AirDrop.

Embrace the filePath argument for usdz_converter

Passing the nodes and their various textures to usdz_converter manually via the command line is frustratingly cumbersome and near impossible to get right every time. Luckily, usdz_converter now includes the -f parameter which accepts a text file to replicate those arguments. For example:

xcrun usdz_converter ./abc/spinner.abc ./usdz/spinner.usdz -f usdzargs.txt

As Nelson Chang covers in this post, it helps to keep your text file open in an editor and modify as needed.

But even creating that file for the first time is a huge pain. If only there was an automated way to do it…

Use the mkargs bash script

PRFTDigitalLabs/c4d-usdz-sample

I’ve thrown together a quick bash script that will generate a text file that can be passed to the -f argument, then run usdz_converter using that file. The script looks through the /baked folder for any files matching the naming pattern that Cinema 4d uses for baked textures then generates the text file. Feel free to fork it and submit pull requests if you have ideas to improve it.

Dan Thompson is Director of Emerging Experiences at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

Adventures in 3D — Streamlining the conversion of assets for iOS Quicklook and USDZ was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

We love USDZ and the AR Quicklook functionality within iOS. We also love Cinema 4d. However, it’s not easy to create a USDZ file directly from Cinema 4d. What follows is a detailed, tech-heavy, walk through of my process creating a Stern Pinball spinner USDZ, starting with Cinema 4d.

A quick note:

USD is a technology created by Pixar. Therefore, it should be no surprise that Maya supports USD better than other tools. Blender, too, has great support for things like glTF and PBR rendering, and is probably an overall better tool for this technology than Cinema 4d. For Blender specific information, I recommend you follow Blender Sushi Guy.

The Plan

I want to create a USDZ file for AR Quicklook on iOS using the 3D tool I am most familiar with: Cinema 4d. It’s most efficient for me to construct the geometry in C4d, do some texture mapping in Body Paint, and deal with the available outputs and conversion utilities to create the final package.

My goals are:

• To take advantage of Augmented Reality by constructing the object at real-world scale.
• To make the object more engaging with animation.
• To explore the process needed to generate and apply PBR materials to the model.
• Avoid using online conversion tools like Vectary or Sketchfab.

The object

I want to create something that has fairly simple, repetitive, mechanical movement. It should have a few different parts and textures. I love pinball, so a pinball part was an obvious choice.

I chose a spinner from a modern Stern Pinball machine.

In doing some quick googling, I found these solidworks files by Joe of http://homebrewpinball.blogspot.com/. Using these will save some time, and will prove that we can adapt a cad-like file format to USDZ through conversion.

Cinema 4d Studio can import solidworks files natively. Alternatively, you can export the .SLDPRT files to .stl files through Fusion 360. Sketchfab outlines that process here.

Cinema 4d can import .stl files. At that point it’s just a matter of re-arranging the objects and ensuring they are set to the correct scale. In Blue below, I’ve ensured that the project is set to 1mm, and in Green the overall Y-dimension is approximately 60mm.

We’ll address the magenta section in the Animation portion of this post.

Tips for success

• You must nest your objects under a single root node for scale and animations to work correctly. In this case, highlighted in Red, stern_spinner is a null object with the other geometry nested below it.
• That root node determines the position of the object in AR. You want it to be positioned at 0,0,0 with your geometry emanating from it. More than likely, you will need to fiddle with the axis to get it to look right.
• While the screenshot does not show it, I would recommend only a single level of child nodes. At some point in the conversion process (possibly to glTF?), I ran into trouble with animations when there were several levels of objects.
• Your nodes should also be named to be easily converted (and recognized): no dots, spaces, special characters, etc…
• Maybe this is obvious, but you should make all your objects editable. It could help to have a ‘dynamic’ version of a file then save it as a flattened version of the file where conversions are made if you are making edits to the geometry during the conversion process.

Conversion

Now that our polygons are in place, we want to see it on a device.

Looking through Apple’s USDZ video (the only formal ‘documentation’ on the technology at this time), the only tool they mention is usdz_converter, and that it supports .obj or Alembic (.abc) as input.

.obj files do not support animation, so that’s out. Luckily, Cinema 4d has a built-in Alembic (.abc) exporter. That should work.

The conversion roadmap will look like this:

.SLDPRT →.stl→.c4d→.abc→.usdz

The .abc export options I use are captured here:

The important choices are File Format and Scale. Changing the other options didn’t seem to have much impact in my use case. The animation options are important, but we’ll deal with those further down.

Now that the .abc has been created, I just needed to run Apple’s conversion tool. Nelson Chang does a great job describing how to use usdz_converter in this post:

USDZ: Converting 3D models and other start-up tips

Verify the conversion

Once the .usdz has been created, OSX tempts you with an opportunity to view and edit properties of the .usdz file in the Xcode SceneKit editor.

Don’t Fall For It!

The textures don’t appear correctly in Xcode. Additionally, you can’t edit the properties for a model in the editor. You can validate the node hierarchy, and potentially material properties, but you can’t effectively edit anything (at this time).

The only sure-fire way to validate the USDZ is by sending it to your phone. This can easily be done with iCloud or AirDrop.

Sadly, you will likely need to do this multiple times as you tweak things (especially materials). I will address a way to optimize this process in my next post.

Animation

For the animation, the goal is to create an infinitely looping spin. This is accomplished through linear interpolation and ensuring that the last frame(+1) is exactly the same as the first frame.

Additionally, the trigger on the switch should appear to be depressed by the spinner arm (this is how points are scored). Each object will have it’s rotation transform adjusted, the bar has it’s position more finely controlled, while the target has a 360° rotation applied over the span of the animation.

Unfortunately, when exporting this animation as .abc, the timing and keyframes don’t always match the source file.

Here is how I resolved this issue:

I created my animation as I wanted them to appear in Cinema 4d. On export to .abc, I set start frame to 0, end frame to 25 (even though the animation is only 24 frames), and ensured the frame step and subframes are set to 1.

Then, I ran through the same usdz_converter process as before, but used the following command line options:

xcrun usdz_converter spinner.abc spinner.usdz -a -l

-a Generate a .usda intermediate file.  Default is .usdc.

-l Leave the intermediate .usd file in the source folder.

This utilizes the options to create an intermediary, ascii .usda file, and does not delete it.

At that point, I opened the .usda file with a text editor and edited the properties that made sense, such as:

endTimeCode = 23
framesPerSecond = 90
startTimeCode = 0
timeCodesPerSecond = 90

Most importantly, however, is the endTimeCode. I essentially ‘trimmed’ the animation so that it looped correctly by setting the value to the final frame that I wanted, then sped it up by changing the framesPerSecond and timeCodesPerSecond values.

I suspect that it is sub-optimal to have the ‘trimmed’ frames/timeSamples remain in the .usda file, so this should be investigated further.

once edited, the .usda file can be converted to the .usdz file thusly:

xcrun usdz_converter spinner.usda spinner.usdz

So, really, the conversion roadmap ends up being:

.SLDPRT →.stl→.c4d→.abc→.usda→.usdz

phew.

Applying Materials / Texturing

I’m not going to describe the whole process of texturing objects in Cinema 4d. If you need help getting started, Greyscalegorilla will have what you need.

If you have any tips for approximating PBR in C4d to match AR Quicklook, I would love to hear your feedback.

If you’re importing 3rd party materials, or you are applying your own baked materials, I found this post extremely helpful. It does a great job explaining PBR, and where each texture should be applied.

In my brief experience, I wasn’t able to get C4d to 100% accurately render the materials the same way as AR Quicklook, even with the Physical Renderer in Cinema 4d Studio. However, I was able to approximate things to a point that I was able to implement a process that worked well for me for exporting materials from C4d:

Exporting Texture Channels

I used UV mapping for all of my materials. Often times I had to manually manipulate those UVs, particularly for the brushed metal effect on the bracket.

To keep your textures and .usdz in sync, remember to re-export your .abc file after you make changes to your UVs in Cinema 4d.

The most efficient process for me was to bake objects with the relevant channels selected:

Baking

By selecting your object, then selecting Objects>Bake Object, you will be presented with this screen:

By baking all of the objects, with the relevant channels selected, then deleting any extraneous textures, you will have a uniformly named set of textures, prefixed by their node:

barAmbient_Occlusion.png
barMetallic.png
barNormal.png
barRoughness.png
barSurface_Color.png
bracketMetallic.png
bracketRoughness.png
bracketSurface_Color.png
switchAmbient_Occlusion.png
switchMetallic.png
switchNormal.png
switchRoughness.png
switchSurface_Color.png
targetMetallic.png
targetRoughness.png
targetSurface_Color.png

Alternatively, you can use the Bake Texture tag on an object for more granular control of the channels that you are exporting. It’s output can be configured to match the naming pattern above.

Finally, once you have all of your textures prepared, you are able to pass those nodes and file names into usdz_converter as Nelson Chang describes here.

Wow, what a ride. This is a lot of work, especially if you need to validate very minor changes to your model or textures. Luckily, my next post offers some tips, tricks, and a tool on how to automate this process.

Dan Thompson is Director of Emerging Experiences at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

Adventures in 3D — A Cinema 4d to iOS AR Quicklook / USDZ Workflow was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

The promise — Why use USDZ?

What do USDZ files provide that make them different? Why would you use them? How big is that impact?

Without an experienced group like Perficient Digital Labs, deploying an Augmented Reality experience can be very technical. You may need knowledge of multiple frameworks, limitations of platforms, and the compatibility issues between those technologies. Similarly, development for AR requires a high level of design finesse. Poor user experience or obtrusiveness can ruin even the best idea. Primarily, the experience has to provide value, even if it’s just entertainment. It’s become vitally important that the technologies we use account for these values and support creators in a helpful way.

When making an AR app, you would normally start with a designer making 3D assets. Those assets are eventually handed over to a developer. To use them, that dev needs to import everything, potentially manipulate and map textures, then ensure the animations for those assets are working correctly. This handoff is taxing and time-consuming. A lot can go wrong and many developers aren’t well-versed in the intricacies of 3D asset production.

iOS Quicklook by way of the USDZ file format feels like a solution that resolves these issues. It promotes a plug and play format that makes life easier for both developers, designers, and consumers (as long as they’re using iPhones).

A USDZ file is a single file containing all of the needed assets. Essentially it’s a .zip file that has the geometry, animation, and textures all in one convenient bundle. Once it’s prepared by a designer, a developer can just run with it. It’s a crisp, lightweight, animated model that’s ready to go right away. It’s the type of workflow that saves devs sleep and designers headaches.

About iOS…

Unfortunately, right now, iOS is the only consumer device platform that natively supports USDZ. All the considerations that go into developing an application for iOS, Android, Web, or a mixture of the three apply to USDZ as well. It’s very unfortunate that Apple didn’t choose to utilize the glTF file format, as it’s already more established and supported on many more platforms.

But the UX on iOS is top notch. When viewed on an iPhone, Quicklook provides a great experience. It allows a user to quickly view the model in 3D or in AR. The quality of the assets are spectacular. Take a photo or video to share, or share the asset itself via iMessage or email. It’s perfect for viewing an inventory of parts or products. USDZ files can be linked to from the web, allowing users to view objects in 3D / AR just by clicking a link on their iPhone. This is how it should be, but time will tell to see if companies take the time to implement USDZ on their web sites or applications.

How to get started

The golden use case for iOS Quicklook with USDZ is to add 3D / AR previews to a set of objects. It could be an online store, a gallery, a manufacturer of parts, etc… Were the user to benefit from viewing the objects in true-to-life scale in augmented reality, ARKit provides great space detection and lighting estimation to allow a user to visualize the object in real space, then share that view with others.

For an example, check out Apple’s sample AR Quicklook Gallery.

Adding AR Quicklook functionality to a new or existing online gallery is quite easy. In most cases it’s simply a matter of adding one or two attributes to existing links within that platform.

If this use case makes sense, you need to go about creating (or converting existing) assets. Converting existing assets to USDZ takes some experience, but can be done. The example above was easily converted from a solidworks file.

Creating new assets can be done through a 3D scanning or Photogrammetry process of real-life inventory. Another option would be to create virtual replicas with 3D modeling and texturing tools. In any case, you’ll want to consult with someone familiar with the tools and techniques required to create great looking assets.

The power of QR codes

Finally, a concept that hasn’t been talked about much is the ability to quickly link to a USDZ asset via a QR code.

This is magical for two reasons:

1. USDZ files can be hosted on the web, just like any other multimedia file.
2. iOS users can scan QR codes from the Camera app. When a user follows that link, it automatically opens in Safari (which is the only browser that can handle USDZ files. This is a pretty direct flow for an iPhone user wanting to view an object in Augmented Reality.

Dan Thompson is Director of Emerging Experiences at Perficient Digital Labs. Read more about how Perficient Digital Labs uses Emerging Technology to create innovative experiences that transform the way our clients do business here.

Adventures in 3D — Why and How you should be investing in iOS Quicklook and USDZ was originally published in Perficient Digital Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.