Diego Prilusky: How volumetric video brings a new dimension to filmmaking

Recorded atJuly 20, 2019
EventTEDSummit 2019
Duration (min:sec)11:30
Video TypeTED Stage Talk
Words per minute148.6 very slow
Readability (FK)48.53 difficult
SpeakerDiego Prilusky

Official TED page for this talk


In this talk and tech demo, filmmaker Diego Prilusky introduces the next chapter in moviemaking: volumetric video, a 360-degree experience powered by hundreds of cameras that capture light and motion from every angle. Check out how this technology creates a new type of immersive storytelling -- and catch Prilusky's reshoot of an iconic dance number from "Grease" captured with volumetric video.

Text Highlight (experimental)
100:12 I love making movies.
200:17 Motion pictures have been in existence for more than a hundred years.
300:22 Filmmaking hasn't changed for the dimensional mindset.
400:27 Placing the camera in a scene and pressing "record" hasn't changed.
500:31 Filmmaking is still a frontal experience,
600:34 and creating the film has the possibility
700:37 to follow the same direction of the content creation.
800:42 We still stand in front of a flat image,
900:45 watching the fiction.
1000:48 There's nothing wrong with it.
1100:49 I love watching movies and going to the theaters.
1200:52 The experiences can be such emotional experiences.
1300:55 The art and craft of emotional experiences within a frame
1400:59 can be so strong to drive a stronger emotion.
1501:03 The question we're asking is,
1601:05 How the experience of motion pictures can exist beyond the flat screen.
1701:10 How can we start creating content
1801:12 for the next generation of content experiences?
1901:16 Traditionally, when we imagine a scene,
2001:19 we look at the frame and the composition.
2101:22 We have to think about how we create depth and parallax
2201:25 using foreground, background elements as the camera moves.
2301:30 With the technology today and devices of VR glasses,
2401:33 AR glasses, smart devices,
2501:36 allowing three-dimensional and full navigation in space,
2601:40 we have the possibility to enable audiences
2701:43 to experience content from multiple perspectives.
2801:46 What we have to think about is how we take this technology,
2901:50 all the capabilities,
3001:51 and enable the experience to move farther away inside the scene.
3101:57 Now we're not talking about video games or computer-generated actors,
3202:01 which look tremendously realistic.
3302:04 We're talking about real actors and real performance,
3402:07 performing onstage.
3502:10 We have to start thinking how we capture the actors
3602:14 and how we capture the real scene
3702:16 in order to immerse inside.
3802:18 Now, we're familiar with the 360-degree video,
3902:21 where you place a camera inside the scene
4002:24 and you can create this beautiful panoramic image all around you,
4102:28 but from the same aspect,
4202:29 filmmaking is still frontal.
4302:32 In order to emerge fully inside the scene,
4402:35 we will need to capture the light from all the possible directions.
4502:39 We will have to surround the scene with an enormous amount of sensors,
4602:44 with all possible capabilities to capture the light
4702:47 and enable us to emerge inside afterwards again.
4802:51 Now, in this setup,
4902:53 there's no more foreground or background
5002:55 or a camera placed in space
5102:57 but hundreds of sensors capturing the light
5203:01 and capturing the motion from all the possible directions.
5303:05 With the new technological advancements,
5403:07 we can start looking at 3D photography,
5503:09 capturing the light from multiple perspectives,
5603:12 enabling us to reconstruct the object.
5703:15 This is like photography in 3D space.
5803:18 Now, with these technological advancements,
5903:21 we can record video not just as a flat image
6003:24 but as a volume.
6103:26 This is what we call "volumetric video,"
6203:29 and it has the capability to record every action of the scene
6303:32 as a full three-dimensional volume.
6403:35 Now, what is a voxel?
6503:38 A voxel is like a three-dimensional pixel,
6603:41 but instead of being a flat image square, staying light and colored,
6703:46 it's like a three-dimensional cube in space,
6803:49 with x, y, and z positions.
6903:53 This enables us to create a full capture of the scene
7003:56 from any perspective.
7103:58 Now this renders a fully light-immersive scene
7204:03 from multiple perspectives.
7304:06 This capability requires an insane amount of information to be processed.
7404:10 We will have to capture the light from an enormous amount of cameras
7504:15 to create this information.
7604:18 Now, in order to do such a thing,
7704:21 we would need a setup that would host a numerous amount of cameras
7804:28 installed in a stage
7904:30 and a stage big enough in order to fit a full cinematic experience.
8004:36 Now that sounds like a crazy idea, but that's exactly what we did.
8104:41 For the last three years,
8204:43 we have been building a huge volumetric camera chamber.
8304:48 It's 10,000 square feet of a stage,
8404:51 enabling to capture the action from any location.
8504:55 We have deployed hundreds of cameras,
8604:57 sending a tremendous amount of information
8704:59 to a huge data center powered by Intel supercomputers.
8805:04 The ability to have this 10,000 feet
8905:08 enables us to fit any kind of action,
9005:11 any kind of performance.
9105:13 It is the size of an average Broadway stage.
9205:17 We call it Intel Studios,
9305:19 and it's the largest volumetric stage in the world,
9405:22 with the objective of enabling and exploring
9505:26 the next generation of this immersive media filmmaking.
9605:30 Now, to test these ideas,
9705:33 we were thinking about what we can do as the first scene to try it out.
9805:39 So we chose the Western scene.
9905:42 We brought horses, set designers, dirt,
10005:46 everything needed to create the full scene of a Western.
10105:50 But this time, there was no camera inside.
10205:54 There was nothing really moving besides all the cameras
10305:58 installed outside.
10406:00 The challenge of the actors was tremendous.
10506:03 They have to perform a flawless action visible from all the directions.
10606:09 There's no possibility to hide a punch or not show the action.
10706:14 Everything is captured and everything is seen.
10806:17 The output of the capture --
10906:21 this is our future capture --
11006:22 opened our eyes for the immense capabilities.
11106:25 It was like a full 3D scan of the entire scene.
11206:29 We were able to move around and travel in the space.
11306:33 The thing about this,
11406:34 it's not anymore about perceiving the light emitted from a screen
11506:38 but now traveling inside the light,
11606:41 traveling inside the scene.
11706:42 This obviously opens possibilities for an enormous amount
11806:45 of storytelling and methodologies of creation.
11906:49 This is the possibilities of your personal narrative,
12006:53 the possibility of creating your own story inside,
12106:56 or maybe following other stories.
12206:59 Let's take a look at one of the last renders and see.
12307:01 (Music)
12407:03 What you're seeing here is full volumetric video,
12507:06 and there's no physical camera in the scene.
12607:08 (Music)
12707:10 We have the full control
12807:11 (Music, sounds of combat)
12907:13 of space and time.
13007:15 (Music, sounds of combat)
13107:26 Now, again, no physical camera was here.
13207:29 Everything was captured surrounding.
13307:31 Now, this is very nice,
13407:34 but what if we wanted to see the scene, maybe, from the eyes of the horse?
13507:38 Well, we can do that as well.
13607:41 (Horse galloping)
13707:42 So what you're seeing right now is the same action,
13807:45 but this time, we're watching exactly from the eyes of the horse.
13907:49 The possibilities are, well, unlimited.
14007:53 (Applause)
14107:54 Thank you.
14207:55 (Applause)
14307:58 So this is all great for creators and storytellers.
14408:03 It really opens a huge canvas
14508:05 for a different type of storytelling and moviemaking.
14608:08 But what about the audience?
14708:09 How can the audience experience this differently?
14808:12 In order to [create] our explorations,
14908:14 we partnered with Paramount Pictures
15008:16 in order to explore immersive media in a Hollywood movie production.
15108:21 Together with the director Randal Kleiser,
15208:23 we reimagined the iconic movie of 1978,
15308:28 "Grease."
15408:29 Some of you know it, some of you don't.
15508:31 A 40-year-old movie, amazing experience.
15608:35 And our goal was really to look at how we can take this iconic action and dance
15708:41 and bring it deeper into the experience,
15808:44 bring it deeper into the audience.
15908:46 Imagine that you can not just watch the movie
16008:49 but get inside it and dance with the actors
16108:52 and dance with the performance.
16208:54 Now we're breaking, really, the traditional 2D mindset of thinking,
16309:00 and bringing a much richer possibility of moviemaking
16409:03 and content creation.
16509:05 But why watch it on the screen?
16609:08 Let's try to bring these actors here on the stage.
16709:17 So they're not going to really come --
16809:19 I'm going to use an iPad.
16909:20 (Laughter)
17009:21 Sorry.
17109:22 I'm going to use an iPad in order to bring in augmented reality.
17209:27 Now, obviously, these devices have their own limitations
17309:33 in terms of the data-computing process,
17409:35 so we have to reduce the amount of resolution.
17509:39 So what I'm doing now, I'm placing here a marker,
17609:42 so I'll be able to position exactly where I want everyone to appear.
17709:54 OK.
17809:55 I think we have them here.
17909:57 (Applause)
18010:04 John Travolta, or --
18110:05 (Laughter)
18210:06 a version of him.
18310:08 Let's take a look.
18410:10 (Video) Female: Hey.
18510:11 Male: And that is how it's done.
18610:13 Female: Your turn.
18710:14 Male: Hey, guys! Check this out.
18810:16 (Song: "You're the one that I want")
18910:19 Danny: Sandy!
19010:20 Sandy: Tell me about it, stud.
19110:24 (Singing) I got chills. They're multiplying
19210:28 And I'm losing control
19310:32 'Cause the power you're supplying
19410:37 It's electrifying!
19510:39 (Video ends)
19610:40 (Applause and cheers)
19710:45 Diego Prilusky: Thank you.
19810:47 (Applause and cheers)
19910:49 So as you can see,
20010:52 we can watch and experience content in the traditional way
20110:55 or in an immersive way.
20210:58 Really, the possibilities are open.
20311:00 We're not trying to change or replace movies.
20411:04 We're enhancing them.
20511:06 The technologies enable new possibilities to start thinking beyond the flat screen.
20611:12 We're in immersive and really exciting times in filmmaking.
20711:16 We're at the threshold of a new era.
20811:18 We're opening the gates for new possibilities
20911:22 of immersive storytelling,
21011:25 and exploration and defining what immersive media filmmaking means.
21111:30 We're really just at the beginning,
21211:33 and we invite you all to join us.
21311:36 Thank you.
21411:37 (Applause)