Improved model rendering, lighting effects, and animation techniques have allowed these experiences to show users a whole new augmented world through the windows of their phones. But these experiences have only recently been able to pay off one of the most compelling parts of the “real world”: creating shared experiences. The next generation of AR will break out of a single user’s phone to be a truly shared experience where users can see, interact with, and alter each other’s realities. Recently one of our lead developers put some of these new capabilities to the test to create a multiplayer AR experiment: a multiplayer basketball game where two or more players could shoot baskets a’la old-school carnival games.

Although both Apple and Google have features in their AR platforms (ARKit and ARCore respectively) supporting shared experiences, the concept is still relatively new — so new that there isn’t a firm name for it yet. The idea is talked about in terms such as “social AR”, “multiuser AR”, “AR Cloud”, “peer-to-peer AR”, and “shared AR”. We like to think of this as a “multiuser AR” challenge — could we make an app that allowed users on separate devices to play in the same AR space and have their actions affect one another in consequential ways?

The primary challenge of a multiuser AR experience is establishing a shared location coordinate set. We need to place objects in the real world for a single user that would also show up in the same places for other users. Positioning of AR objects (both stationary and also those in motion/collision) requires two users to have coordinate systems relatively identical to each other. Currently Apple’s AR framework, ARkit is more robust than Google’s ARCore and offers more opportunities to test these capabilities, so we chose to develop an iOS app.

As a first step, we needed to get a shared anchor coordinate for each device to be spatially aware of. So we used an AR mainstay: an AR Marker. In its most simple form, an AR Marker can be a symbol printed on a piece of paper and stuck to a wall. But through machine learning, most distinct objects can act as a marker. A marker in multiuser AR allows each device to relate objects relative to the same known position within the real world space. For our experiment, we used a marker on a wall to serve as our AR basketball goal — a focal point for our game both in terms of shared gameplay and relative coordinate positioning.

If we were going to have more than one person shoot a basketball at this goal, we had to tackle a more complex problem: having both user actions and objects share their changing, real-time positions between devices. If I shoot a basketball at the same time as my friend, then I should see both of our balls flying through the real world. For this, we relied on Apple’s MultipeerConnectivity Framework. This allows two devices sharing the same AR experience to pass data back and forth via peer-to-peer connections. Using this responsive data handler, we were able to let two or more players shoot basketballs simultaneously, each seeing the shots in real-time.

Now that we had players able to shoot balls in the same shared AR space, we quickly realized that objects couldn’t just be aware of each, they had to react to each other in realistic ways. Luckily Apple has provided a 3D object physics engine within its SceneKit Framework. With each user’s device aware of ball sizes and positions, adding collision detection and gravity effects wasn’t very difficult. It allows balls to bounce off each other in midair, knock each other into walls, block shots and more.

In the end, these multiuser additions helped make AR feel even more real than other single-user experiences that we have created. Seeing a basketball goal drop into the middle of your office looks cool, for sure … but being trapped on a single user’s phone makes the AR feel more like a novelty than something truly engaging. Alternatively, seeing coworkers react to each other’s actions in a shared AR basketball court was tangibly impactful, connecting people rather than isolating them. The experience, although framed in users’ phones, broke free of devices, augmenting the real world with laughter, playfulness, and engagement — the exact ingredients that both brands and consumers crave.