Doug Roble: Digital humans that look just like us

Recorded atApril 15, 2019
Duration (min:sec)12:21
Video TypeTED Stage Talk
Words per minute159.51 slow
Readability (FK)68.06 very easy
SpeakerDoug Roble

Official TED page for this talk


In an astonishing talk and tech demo, software researcher Doug Roble debuts "DigiDoug": a real-time, 3-D, digital rendering of his likeness that's accurate down to the scale of pores and wrinkles. Powered by an inertial motion capture suit, deep neural networks and enormous amounts of data, DigiDoug renders the real Doug's emotions (and even how his blood flows and eyelashes move) in striking detail. Learn more about how this exciting tech was built -- and its applications in movies, virtual assistants and beyond.

Text Highlight (experimental)
100:13 Hello.
200:15 I'm not a real person.
300:17 I'm actually a copy of a real person.
400:19 Although, I feel like a real person.
500:22 It's kind of hard to explain.
600:24 Hold on -- I think I saw a real person ... there's one.
700:28 Let's bring him onstage.
800:33 Hello.
900:35 (Applause)
1000:40 What you see up there is a digital human.
1100:43 I'm wearing an inertial motion capture suit
1200:46 that's figuring what my body is doing.
1300:49 And I've got a single camera here that's watching my face
1400:53 and feeding some machine-learning software that's taking my expressions,
1500:58 like, "Hm, hm, hm,"
1601:02 and transferring it to that guy.
1701:05 We call him "DigiDoug."
1801:09 He's actually a 3-D character that I'm controlling live in real time.
1901:16 So, I work in visual effects.
2001:19 And in visual effects,
2101:20 one of the hardest things to do is to create believable, digital humans
2201:26 that the audience accepts as real.
2301:28 People are just really good at recognizing other people.
2401:32 Go figure!
2501:35 So, that's OK, we like a challenge.
2601:39 Over the last 15 years,
2701:40 we've been putting humans and creatures into film
2801:45 that you accept as real.
2901:48 If they're happy, you should feel happy.
3001:51 And if they feel pain, you should empathize with them.
3101:58 We're getting pretty good at it, too.
3202:00 But it's really, really difficult.
3302:03 Effects like these take thousands of hours
3402:07 and hundreds of really talented artists.
3502:10 But things have changed.
3602:13 Over the last five years,
3702:14 computers and graphics cards have gotten seriously fast.
3802:20 And machine learning, deep learning, has happened.
3902:25 So we asked ourselves:
4002:27 Do you suppose we could create a photo-realistic human,
4102:31 like we're doing for film,
4202:33 but where you're seeing the actual emotions and the details
4302:39 of the person who's controlling the digital human
4402:43 in real time?
4502:45 In fact, that's our goal:
4602:47 If you were having a conversation with DigiDoug
4702:51 one-on-one,
4802:53 is it real enough so that you could tell whether or not I was lying to you?
4902:59 So that was our goal.
5003:02 About a year and a half ago, we set off to achieve this goal.
5103:06 What I'm going to do now is take you basically on a little bit of a journey
5203:10 to see exactly what we had to do to get where we are.
5303:15 We had to capture an enormous amount of data.
5403:20 In fact, by the end of this thing,
5503:23 we had probably one of the largest facial data sets on the planet.
5603:28 Of my face.
5703:29 (Laughter)
5803:32 Why me?
5903:33 Well, I'll do just about anything for science.
6003:36 I mean, look at me!
6103:38 I mean, come on.
6203:43 We had to first figure out what my face actually looked like.
6303:49 Not just a photograph or a 3-D scan,
6403:52 but what it actually looked like in any photograph,
6503:56 how light interacts with my skin.
6603:59 Luckily for us, about three blocks away from our Los Angeles studio
6704:05 is this place called ICT.
6804:07 They're a research lab
6904:09 that's associated with the University of Southern California.
7004:12 They have a device there, it's called the "light stage."
7104:16 It has a zillion individually controlled lights
7204:20 and a whole bunch of cameras.
7304:22 And with that, we can reconstruct my face under a myriad of lighting conditions.
7404:29 We even captured the blood flow
7504:31 and how my face changes when I make expressions.
7604:35 This let us build a model of my face that, quite frankly, is just amazing.
7704:41 It's got an unfortunate level of detail, unfortunately.
7804:45 (Laughter)
7904:47 You can see every pore, every wrinkle.
8004:50 But we had to have that.
8104:52 Reality is all about detail.
8204:55 And without it, you miss it.
8304:58 We are far from done, though.
8405:01 This let us build a model of my face that looked like me.
8505:05 But it didn't really move like me.
8605:08 And that's where machine learning comes in.
8705:11 And machine learning needs a ton of data.
8805:15 So I sat down in front of some high-resolution motion-capturing device.
8905:20 And also, we did this traditional motion capture with markers.
9005:25 We created a whole bunch of images of my face
9105:28 and moving point clouds that represented that shapes of my face.
9205:33 Man, I made a lot of expressions,
9305:36 I said different lines in different emotional states ...
9405:40 We had to do a lot of capture with this.
9505:43 Once we had this enormous amount of data,
9605:46 we built and trained deep neural networks.
9705:51 And when we were finished with that,
9805:52 in 16 milliseconds,
9905:55 the neural network can look at my image
10005:58 and figure out everything about my face.
10106:02 It can compute my expression, my wrinkles, my blood flow --
10206:07 even how my eyelashes move.
10306:10 This is then rendered and displayed up there
10406:13 with all the detail that we captured previously.
10506:18 We're far from done.
10606:20 This is very much a work in progress.
10706:22 This is actually the first time we've shown it outside of our company.
10806:25 And, you know, it doesn't look as convincing as we want;
10906:29 I've got wires coming out of the back of me,
11006:32 and there's a sixth-of-a-second delay
11106:34 between when we capture the video and we display it up there.
11206:38 Sixth of a second -- that's crazy good!
11306:41 But it's still why you're hearing a bit of an echo and stuff.
11406:46 And you know, this machine learning stuff is brand-new to us,
11506:50 sometimes it's hard to convince to do the right thing, you know?
11606:54 It goes a little sideways.
11706:56 (Laughter)
11806:59 But why did we do this?
11907:03 Well, there's two reasons, really.
12007:05 First of all, it is just crazy cool.
12107:08 (Laughter)
12207:09 How cool is it?
12307:10 Well, with the push of a button,
12407:13 I can deliver this talk as a completely different character.
12507:17 This is Elbor.
12607:22 We put him together to test how this would work
12707:24 with a different appearance.
12807:27 And the cool thing about this technology is that, while I've changed my character,
12907:32 the performance is still all me.
13007:35 I tend to talk out of the right side of my mouth;
13107:38 so does Elbor.
13207:39 (Laughter)
13307:42 Now, the second reason we did this, and you can imagine,
13407:44 is this is going to be great for film.
13507:47 This is a brand-new, exciting tool
13607:49 for artists and directors and storytellers.
13707:55 It's pretty obvious, right?
13807:56 I mean, this is going to be really neat to have.
13907:59 But also, now that we've built it,
14008:01 it's clear that this is going to go way beyond film.
14108:05 But wait.
14208:07 Didn't I just change my identity with the push of a button?
14308:11 Isn't this like "deepfake" and face-swapping
14408:14 that you guys may have heard of?
14508:17 Well, yeah.
14608:19 In fact, we are using some of the same technology
14708:22 that deepfake is using.
14808:23 Deepfake is 2-D and image based, while ours is full 3-D
14908:28 and way more powerful.
15008:31 But they're very related.
15108:33 And now I can hear you thinking,
15208:35 "Darn it!
15308:36 I though I could at least trust and believe in video.
15408:40 If it was live video, didn't it have to be true?"
15508:44 Well, we know that's not really the case, right?
15608:48 Even without this, there are simple tricks that you can do with video
15708:52 like how you frame a shot
15808:55 that can make it really misrepresent what's actually going on.
15909:00 And I've been working in visual effects for a long time,
16009:03 and I've known for a long time
16109:05 that with enough effort, we can fool anyone about anything.
16209:11 What this stuff and deepfake is doing
16309:13 is making it easier and more accessible to manipulate video,
16409:18 just like Photoshop did for manipulating images, some time ago.
16509:25 I prefer to think about
16609:26 how this technology could bring humanity to other technology
16709:31 and bring us all closer together.
16809:34 Now that you've seen this,
16909:36 think about the possibilities.
17009:39 Right off the bat, you're going to see it in live events and concerts, like this.
17109:45 Digital celebrities, especially with new projection technology,
17209:50 are going to be just like the movies, but alive and in real time.
17309:55 And new forms of communication are coming.
17409:59 You can already interact with DigiDoug in VR.
17510:03 And it is eye-opening.
17610:05 It's just like you and I are in the same room,
17710:09 even though we may be miles apart.
17810:12 Heck, the next time you make a video call,
17910:15 you will be able to choose the version of you
18010:18 you want people to see.
18110:20 It's like really, really good makeup.
18210:24 I was scanned about a year and a half ago.
18310:29 I've aged.
18410:30 DigiDoug hasn't.
18510:32 On video calls, I never have to grow old.
18610:38 And as you can imagine, this is going to be used
18710:41 to give virtual assistants a body and a face.
18810:44 A humanity.
18910:45 I already love it that when I talk to virtual assistants,
19010:48 they answer back in a soothing, humanlike voice.
19110:51 Now they'll have a face.
19210:53 And you'll get all the nonverbal cues that make communication so much easier.
19311:00 It's going to be really nice.
19411:01 You'll be able to tell when a virtual assistant is busy or confused
19511:05 or concerned about something.
19611:09 Now, I couldn't leave the stage
19711:12 without you actually being able to see my real face,
19811:14 so you can do some comparison.
19911:18 So let me take off my helmet here.
20011:20 Yeah, don't worry, it looks way worse than it feels.
20111:25 (Laughter)
20211:29 So this is where we are.
20311:30 Let me put this back on here.
20411:32 (Laughter)
20511:35 Doink!
20611:37 So this is where we are.
20711:39 We're on the cusp of being able to interact with digital humans
20811:43 that are strikingly real,
20911:45 whether they're being controlled by a person or a machine.
21011:49 And like all new technology these days,
21111:54 it's going to come with some serious and real concerns
21211:59 that we have to deal with.
21312:02 But I am just so really excited
21412:04 about the ability to bring something that I've seen only in science fiction
21512:09 for my entire life
21612:11 into reality.
21712:13 Communicating with computers will be like talking to a friend.
21812:18 And talking to faraway friends
21912:20 will be like sitting with them together in the same room.
22012:24 Thank you very much.
22112:26 (Applause)