Seman­tic tech­no­lo­gies for com­plex metadata

How the dwerft rese­arch alli­ance is advan­cing struc­tu­ral chan­ge in the film and TV industry


Has­n’t the film world been at the fore­front of digi­ti­sa­ti­on with its digi­tal visu­al effects and strea­ming plat­forms sin­ce works like Ava­tar or Incep­ti­on and plat­forms à la Net­flix? But even if we have long sin­ce said good­bye to 35 mm and can crea­te enti­re gala­xies with bits and bytes: the pro­duc­tion chain in the back­ground is any­thing but digital.Research fin­dings to be pre­sen­ted at the Media­Tech Hub Conference

The rese­arch pro­ject dwerft - lin­ked meta­da­ta for media wants to chan­ge this and has been rese­ar­ching meta­da­ta and arti­fi­ci­al intel­li­gence in the film and tele­vi­si­on indus­try at its Babels­berg loca­ti­on for the past three years. The rese­arch pro­ject for inno­va­ti­ve media tech­no­lo­gies, which was fun­ded by the Fede­ral Minis­try of Edu­ca­ti­on and Rese­arch, is now in its final pha­se. The fin­dings and first pro­to­ty­pes will be pre­sen­ted for the first time during the Media­Tech Hub Con­fe­rence in Pots­dam on 10 and 11 November.

“What is going to come after arti­fi­ci­al intel­li­gence? This is the cre­do we adopted,” says dwerft pro­ject coor­di­na­tor Clau­dia Wolf when explai­ning their approach. Arti­fi­ci­al intel­li­gence (AI) is alre­a­dy being appli­ed to finis­hed con­tent and can be trai­ned to search for images, sound sequen­ces and text seg­ments, for exam­p­le. But meta­da­ta is nee­ded in order to be able to ope­ra­te the search algo­rith­ms cor­rect­ly. And the bet­ter this is pre­pared, the bet­ter the AI can eva­lua­te it. It is only then that the audio­vi­su­al value chains can be ful­ly exploi­ted and pro­duc­tion pro­ces­ses well automated.

Meta­da­ta is thus the deter­mi­ning fac­tor when it comes to the future of film dis­tri­bu­ti­on, mar­ke­ting and auto­ma­ti­on. It is rele­vant for the end user when you are tal­king about film sug­ges­ti­ons on strea­ming plat­forms or very spe­ci­fic search queries: roman­tic films taking place on Valentine’s Day? A nice autumn film with sce­nes in Cen­tral Park? Or the best films with a sound­track set to music of the 70s? All this can be pre­pared and dis­play­ed with the right meta­da­ta linking.

Auto­ma­ted pro­duc­tion pro­ces­ses and dis­tri­bu­ti­on thanks to metadata

More important for film pro­duc­tion, howe­ver, is the meta­da­ta which alre­a­dy exists during shoo­ting and pre-pro­­duc­­ti­on. A script by its­elf alre­a­dy con­ta­ins infor­ma­ti­on about actors, cos­tu­mes, times, wea­ther and loca­ti­ons, but this infor­ma­ti­on is no lon­ger available as pro­ces­sed data later on in the pro­duc­tion and often gets lost. This meta­da­ta has to be pre­pared cor­rect­ly and syn­chro­no­us­ly during the pro­duc­tion pro­cess so that it can be acces­sed at a later stage for the appro­pria­te deployment.

Clau­dia Wolf: “The­re are curr­ent­ly data silos in the pro­duc­tion pha­ses that should actual­ly be lin­ked tog­e­ther. The dwerft solu­ti­on aims at sto­ring meta­da­ta during the actu­al pro­duc­tion and ensu­ring that the­re is a sui­ta­ble value chain for this data. Our goal is to have the best pos­si­ble pre­pa­ra­ti­on so that it can be made available to all acti­vi­ties at any time.”

The meta­da­ta of sound­tracks, images, cos­tu­mes, etc. are recor­ded, cate­go­ri­sed and aggre­ga­ted. Tech­ni­cal lin­king is nee­ded in order to be able to real­ly make use all of this addi­tio­nal infor­ma­ti­on. To this end, the dwerft pro­ject has deve­lo­ped struc­tures and rese­ar­ched seman­tic lin­king. The par­ti­ci­pa­ting part­ners - DRA Stif­tung Deut­sches Rund­funk­ar­chiv, film­wer­te GmbH, Inter­la­ke Media GmbH, Rotor Film and trans­fer­me­dia pro­duc­tion ser­vices - have con­tri­bu­ted their exper­ti­se and part­ly their own soft­ware solu­ti­ons: the Deut­sches Rund­funk­ar­chiv for archi­ving, film­wer­te with video-on-demand, or Rotor Film with its own plug-in for post-pro­­duc­­ti­on. The Lin­ked Media Data Cloud is the core tech­no­lo­gy pro­vi­ding the basis for other soft­ware solu­ti­ons. The data is map­ped the­re so well that it can be acces­sed depen­ding on the pur­po­se - in other words, a know­ledge data­ba­se whe­re a lot of seman­tic ground­work has alre­a­dy been done.

Use and mar­ke­ting of music and social media

The dwerft mana­gers will also be pre­sen­ting a spe­cial case with GEMA at the Media­Tech Hub Con­fe­rence. Film music is an important fac­tor in the pro­duc­tion and explo­ita­ti­on of films. To start with, every music right for a sound­track has to be reques­ted sepa­ra­te­ly from GEMA. This invol­ves manu­al lists being pre­pared and sub­mit­ted. But then the­re are films which often have their own com­po­si­ti­ons, and the­se in turn have to be sent to GEMA for rights explo­ita­ti­on. dwerft was invol­ved in working on enab­ling pro­du­cers to link their source music direct­ly in future to the GEMA data­ba­se which then pro­vi­des them with all of the neces­sa­ry infor­ma­ti­on. Every type of music is lin­ked to a spe­ci­fic time code so that it can be iden­ti­fied at the right place. This is some­thing that does­n’t now have to be done at the end of the pro­duc­tion pro­cess, but runs direct­ly in an auto­ma­ted for­mat during production.

Meta­da­ta during the pro­duc­tion pro­cess also has an influence on future film and tele­vi­si­on pro­duc­tions. So-cal­­led pre­dic­ti­ve ana­ly­tics based on arti­fi­ci­al intel­li­gence auto­ma­ti­cal­ly sug­gest lea­ding actors and loca­ti­ons or make com­pa­ri­sons with simi­lar films and their mar­ke­ting - even befo­re a shoot has taken place. This also includes social media mar­ke­ting. “In the case of film dis­tri­bu­ti­on, a lot of data is gene­ra­ted on social media, for exam­p­le, about direc­tors, actors, sto­ries or loca­ti­ons. With the data­ba­se, our dwerft onto­lo­gy, we link them tog­e­ther and give them a struc­tu­red form,” says Clau­dia Wolf. dwerft will also be joi­n­ed by coope­ra­ti­on part­ners on the panel at the MTH Con­fe­rence in Novem­ber 2021 to pre­sent such a case stu­dy as well as other busi­ness pro­ces­ses asso­cia­ted with the Lin­ked Media Data Cloud.

With the fun­ding pha­se now coming to an end, the rese­arch pro­ject is going out into the big wide world and will deve­lop the indi­vi­du­al pro­to­ty­pes to mar­ket matu­ri­ty. Among other things, the dwerft alli­ance will be one of the exhi­bi­tors in the Ger­man Pavi­li­on at the IBC in Ams­ter­dam from 3 to 6 Decem­ber, 2021.

About MTH Blog

The media tech­no­lo­gies of the future are alre­a­dy being used today – not only in the enter­tain­ment sec­tor, but also in a wide varie­ty of indus­tries. Chris­ti­ne Lentz meets up with tech enthu­si­asts, estab­lished com­pa­nies and rese­ar­chers for our month­ly Media­Tech Hub Pots­dam blog to tell the sto­ries behind the inno­va­ti­ve busi­ness models.

More blog articles

About MTH Blog

The media technologies of the future are already being used today – not only in the entertainment sector, but also in a wide variety of industries. Christine Lentz meets up with tech enthusiasts, established companies and researchers for our monthly MediaTech Hub Potsdam blog to tell the stories behind the innovative business models.