How can we combine multimedia data sources such as Google Images to creatively paint pictures for a story that has been generated by a computer? In the Blofeld group at the computational creativity code camp 2015, we got inspired by this question.
Members of Blofeld group are:
1- Philipp Wicke (firstname.lastname@example.org)
2- Robert Pfeiffer (email@example.com)
3- Kasia Bigaj (firstname.lastname@example.org)
4- Hamid Reza Ghaeini (Ghaeini@acm.org)
Our idea is to have multimedia content by using the Google API. The actual process consists of these four steps:
- Telling a story.
- Extracting most salient words
- Putting pictures (representing the salient words/concepts) on a canvas.
- The story determines the scaling/format/position of the pictures.
I this report our focus is on the first part. First of all, we used the non-official characterization list (NOC) to find two actors of our story by some randomized algorithm which tries to find two related actors at random. This is how our story begins! Then from the Scealextric knowledge base, we chose some idiomatic actions that are somehow related to those two actors. Here is where our question arises: Which methods do we have to apply to retrieve the most salient feature of a story in order to find an apt visual representation i.e. an image via Google? To find a correlation between our idiomatic actions of our actors we proposed to use some deterministic approaches. Then we will finish our story by using the story ending knowledge base.
How could we be able to extract the most salient features of a story? They were already provided during the generation of the story. So we could easily use the memory of the story teller algorithm to crawl the internet for a corresponding image. Let’s say our story was about James Bond, we would ask the Google API to search for James Bond images. It all worked out pretty nice until the point where we had to ask Google. Unfortunately we could not use the API and went for Imgur which turned out to be a bad idea. The images are not correctly annotated and the outcome for our search queries was pretty bad.
All codes are available at:
An example story is:
“Eliot Ness paid well and expected absolute loyalty in return. Rick Deckard finally caught up to Eliot Ness. Rick Deckard finally tracked Eliot Ness down. Thereafter Rick Deckard kept Eliot Ness on a very tight leash indeed; Eliot Ness was never truly free again.”