Institute For The Future

Podcast: What's the meaning of 'When Everything is Media'?

A discussion with Bradley Kreit, research director at IFTF for the When Everything is Media  research

Rapid advances in network speed, artificial intelligence, immersive media platforms, and the Internet of Things are transforming our communications experiences and leading to a future where everything is media. In this episode of the Institute for the Future podcast, IFTF Research Director Bradley Kreit discusses the technologies and societal forces that are transforming when, where, how, and why we communicate in a world of ambient media.

Subscribe to the IFTF podcast on iTunes | RSS | Soundcloud | Download MP3

TRANSCRIPT:

Your research project for 2016 was titled When Everything is Media. What does that mean?

Our work in 2016 had this framing called When Everything is Media: The future of ambient communications. The big underlying idea is that we're rapidly headed toward a future where anything you can imagine doing, seeing, interacting with has the potential to be digitally mediated in one way or another. It has the potential to be captured, stored, and affected by digital communication streams.

In our work, we asked a couple of key questions. One was "What are the big underlying technological forces and thresholds that are shaping this space?" The second question was "How will people relate to and understand these technologies?"

Our thinking here is shaped by the sense that even as our technologies change very rapidly, the ways we use them and why we use them tend to be much more familiar. They tend to change much more slowly. For me, a good example is the Super Bowl. In previous eras you would just watch the Super Bowl on the television. Now, most people watching it at home are also on Twitter or other social media, so there's two layers of mediation. We're arguing that events like this will continually be mediated by new technologies.

What are the technologies that are going to enable this future?

What's interesting is that conversational interfaces like Amazon Echo, and virtual reality like Oculus and the HoloLens are being hyped now as the next big thing in computing, interfaces, and communications. In many ways they're being hyped because they're good. Of course it's a sales job, but it's not a sales job in the sense that there's just a ton of potential that's rapidly emerging. For us, the task was to take a step back and say, "We need to understand a world where a lot of different communications platforms are all maturing at once, and we need to understand them in different terms and rethink the entire experience. In our work we identified what we called five big capacities of ambient communications.

What are the five capacities?

The first was what we called Embedded. Embedded is the most familiar communications technology story. It's continued improvements in processor speed. It's the rise of the Internet of Things. It's the understanding that communications technologies are getting embedded everywhere. Right now I'm in a conference room. There's my computer. There's a phone. I have my phone in my pocket. I have a Fitbit, so three or four communications devices here, but that what we expect to see is just an ongoing steady increase of the kinds of communications devices that you would have in any given space.

The second one is Illuminated. This is recognizing that having lots more data or lots more connectivity in and of itself is helpful, but on its own is a dead end. Illuminated moves technology from heuristic to optimal. It's about being able to sense patterns, make sense of data, and take lots of different inputs from lots of different devices and technologies and understand what are the optimal pieces of information that need to be directed to a person or to a machine for any given context toward any given person.

The third is what we called Anticipatory. Anticipatory in many ways is what a lot of the tech giants are chasing right now. Google's relatively new CEO, Sundar Pichai, had a great line in his first shareholder letter as CEO of Google saying, "Just a decade ago computing was still synonymous with big computers that sat on our desks. Over time the computer itself will be an intelligent assistant helping you through your day. We'll move from a mobile first to an AI first world."

I think what he's saying is this idea of anticipation, of technology that knows I want a particular piece of news or a particular advertisement for a product before I know that I want it. If you can anticipate my needs before I'm even aware of them, there's just tremendous potential there for all sorts of purposes.

The fourth is what we called Multi-Sensory. Multi-sensory was an attempt to acknowledge that virtual reality, augmented reality, and those kinds of platforms are becoming very compelling, but at the same time, we shouldn't just expect the world of communications to be screen-based anymore. We are moving toward a world where our communication systems and our digital communication systems are embodied in the same way that person-to-person communications are in life. Things like haptics, wearables, and devices you talk to will come into play.

Finally the fifth big capacity is what called Programmable. What we mean by that is the ability to orchestrate lots and lots of computing resources with very little effort. I started thinking about this capacity when I was traveling for work about a year ago, and I was in Sweden looking out at a gorgeous view and trying to Skype home to my two-year-old daughter in San Francisco.

If you've ever tried talking to a two-year-old over an iPad, you know they can't pay attention for more than a couple minutes. She was chasing my wife around the house. My wife was just desperate for a break because here I am traveling internationally, and so I was just looking at my empty living room back in San Francisco. Just mostly out of boredom I said, "Alexa, play Johnny Cash." Within two seconds, I hear Johnny Cash playing back from San Francisco to me in Switzerland over my iPad.

It was amazing to realize used a very simple voice command to orchestrate powerful computing resources that activated something up in the cloud to play music out in San Francisco, and that music was being played back to me. The enormous potential of being able to affect change around the world through a few spoken words was astonishing, and it pointed to this sense that we'll increasingly be able to program our communications flows and connect to things across distance in a way we never have been able to before.

You then developed a number of forecasts based on these five capacities. I'd like you to talk about some of them. Let's start with "machine-orchestrated entertainment." What's that?

Machine-orchestrated entertainment was an acknowledgement that sophisticated neural nets are getting good at taking data in from previous artwork, music, or so on, and recreating either facsimiles or similar kinds of art or creating an entirely new art on their own. There are a couple examples that illustrated this. One was something called The Next Rembrandt, which was a project by this interesting partnership of machine learning experts and art historians.

They trained a machine learning algorithm to analyze, stroke by stroke, Rembrandt's canon of paintings, and then use that to create a very, very convincing new Rembrandt, or what they called the next Rembrandt. When I show people the two images side by side and I ask them to vote on which one's the real Rembrandt, it usually splits 50/50, 40/60. Every once in a while there's an odd group of people who are strong visual artists that have a better sense of which one's the real Rembrandt. Ultimately what this system did was create a pretty convincing new painting from a 17th century artist.

Another very different example that we used to help us think about this is a service called Jukedeck out of the UK. What's amazing about Jukedeck is that I have no musical background to speak of, but I can specify, let's say, 20 seconds of ambient music, or a number of other genres. There are three or four variables you can specify including mood, time, and length. You hit "create track." Within a few seconds you have your own piece of music that's entirely custom, entirely created by this machine-learning service.

It's purpose is to provide background music for semi-professional videos, audiobooks, podcasts, that kind of stuff, but it's an incredible thing to realize that now an amateur with a little bit of money to license one of their songs can get a custom soundtrack the way that any serious professional outfit or major studio might can do now. It signifies that we're moving rapidly toward a world where machines can do creative work and so that any communications or media experience has the potential to be aided by computational tools. These tools have the potential to super charge creativity and enable us to be much more productive, much more creative than we ever have before.

Let's talk about another forecast. Tell me about "Bio Media."

It's obvious that biometric data is becoming part of how we interact with digital communication systems. Right now, bio media is almost exclusively being used in two fields. One is in health. You have applications that are obvious, like fitness trackers. The second is in security or payment. It's using your thumbprint to pay for your groceries at Whole Foods on your iPhone, that kind stuff. One of my favorite examples is from this pair of DJs named Zeds Dead who got interested in the idea of trying to understand, at a very quantitative level, how music affects their fans. They put heart rate monitors and other bio sensors onto their fans and got this realtime reaction of "when we did this with the beat, here's how the crowd responded." It gets you thinking about a world where our communication streams — and this I think you can particularly imagine in the context of music — are constantly responding not just to what it appears like you're thinking but to your underlying biological states.

In this context I've seen demo products of things like headphones with built-in sensors that can guess your heart rate and try to figure out, "Oh, you're exercising, so we'll play music that'll pump you up and help you run faster or harder. We'll change the beat to match your speed." This sense that our underlying biometrics will start to help shape what we get back in return is an interesting space for thinking about how to personalize communications to your individual audience.

One of the interesting things about this is that it allows our bodies to communicate in ways that we normally have not communicated. Our heart rate, perspiration, those kinds of things, are now modes of communications or signals that we can transmit to other people or machines as opposed to just our voices, or body language, or things like that. They're these entirely new modes of communication, and we have just barely scratched the service.

Yeah, absolutely, I think for good and bad in certain ways. There's that famous work that's been done on microfacial expressions — you might look stoic to somebody who's not trained in that world, but to somebody who's trained in microexpressions, they can tell if you're sad or upset or angry even through just very subtle movements that last half a second. The thing that I think people always worry about is, if you trained in it, you have this superpower that let's you interpret people's emotions, but then you feel like your emotions could be interpreted. Some of the most exciting social applications of this are in health and healthcare. I know there's ongoing work around autism and emotion recognition. Many autistic people have trouble recognizing their own emotions, even though their underlying biometrics give good signals that they might be about to get angry, have an outburst, or be upset. There's lots of work to try to figure out how to interpret those signals far enough in advance that a caregiver could step in and prevent an outburst from happening. So you can imagine the spectrum ranging from cool new ways to make a party a little bit more entertaining all the way to helping address intractable challenges in a space like caregiving.

Tell me about the "Shareable Presence" forecast.

A common misunderstanding that virtual reality is yet another personal isolating form of entertainment. If television was the first big thing that isolated us from the world, now you can strap something like a television onto your head and be totally taken out of the reality around you.

There are obviously great personal entertainment applications of virtual reality, but what is more exciting is shareable presence. It's the ability to use tools like virtual and mixed reality to transcend boundaries, to connect socially, personally, and professionally in ways that we never have been able to before. The best way to think about virtual reality is as a fundamentally social technology, one that's about being able to share yourself and share your presence with others.

There are so many good examples. Microsoft has this great demonstration of something they call holoportation where two people with hololenses and 360 degree capture cameras can, in effect, transport a fully realistic holographic rendering of themselves into another room. Facebook has a ton of cool demos that are worth watching, but the important signal to think about is how video Skyping, FaceTime, and all that stuff has driven technological adoption, in particular among older people. They want to FaceTime their grandchildren back home. Sometimes the only way to build a relationship with a grandchild is through an iPad, and so they buy the iPad or they buy the iPhone so they can connect with the grandchild.

Virtual reality is going to have this same feeling and the same experience. A lot of people don't imagine themselves as users or adopters of virtual reality, but when they see the application not as a tool for entertainment, but as a tool to build social connections we'll rush toward it because it opens up so many new potentials.

The other forecast I wanted you to talk about was the "Expanded Sensorium."

It's very closely related to shareable presence, and it's about being able to project yourself anywhere and all over the world. It's the ability to affect physical change in one way or another anywhere in the world. In our IFTF offices we have couple of these telepresence robots called the Beam, made by a company called Suitable Technologies.

You just click a button on the Suitable Technology app and all of a sudden you're not just in a video chat, but you're physically ... you feel like you're in this robot, and you can push it around and drive it. I've had meetings with people where I look to my left and I feel like the robot is too close, and we have like robot personal space issues, which is interesting in and of itself.

Here's another example:about a year ago a Silicon Valley engineer hacked together a little system to connect his Amazon Echo so that it could open up his garage door and enable his Tesla to drive itself out of the garage when he said, "Alexa, ask Kitt to pull the car out of the garage." Then he'd get a message saying, "Your car is ready for you to drive," and he was supposed to get in it.

We're moving toward is a world where you can not only connect to anyone and share information, but you can connect to potentially anything and affect it out in the world. That's going to be a very big shift that we're just beginning to grapple with.

How do you think that this world of ambient media is going to change us both as a species and as a civilization?

What was striking about this work is how these are tools and technologies that enable us to better are enacting our intentions for communications. My in-laws live in Santa Barbara. We're in Berkeley, California, now, so about 300 miles away. We see them from time to time, but not all that often. When I think about how they communicate with my daughter in terms of using an iPad and FaceTime, what's striking is they're doing the same exact thing that my grandparents who lived in New York were doing when they called us on the phone and tried to have a phone conversation with us. They were living out the same intention of building intimacy with a family member, but at the same time the technologies enable my in-laws to do it much more effectively in a much more natural way than my grandparents could. Of course, for my grandparents, they were able to do it in a much more natural way than their parents could and so on.

The biggest finding of this work was that these technologies will enable us to better enact, seek out, and do things that fulfill our intentions for communication. Something like Pokémon GO is early example of the way mobile and communications technologies embedded around us can enable fun experiences to leap off of the screen and into the city. It's about being able to create fun much more effectively.

You can go down the list of intentions we identified—Collaboration, Control, Empathy, Engagement, Fun, Intimacy, Persuasion, Productivity—and realize that these new communications technologies enable us to do what we want to do but much more effectively and in a much better way.