Recorded Future Demos ‘Predictive’ Search Engine

Recorded Future Demos ‘Predictive’ Search Engine

Dec 03


By Tom Simonite
Technology Review
December 2, 2010

Original Link

A startup called Recorded Future has developed a tool that scrapes real-time data from the Internet to find hints of what will happen in the future. The company’s search tool spits out results on a timeline that stretches into the future as well as the past.

The 18-month-old company gained attention earlier this year after receiving money from the venture capital arms of both Google and the CIA. Now the company has offered a glimpse of how its technology works.

Conventional search engines like Google use links to rank and connect different Web pages. Recorded Future’s software goes a level deeper by analyzing the content of pages to track the “invisible” connections between people, places, and events described online.

“That makes it possible for me to look for specific patterns, like product releases expected from Apple in the near future, or to identify when a company plans to invest or expand into India,” says Christopher Ahlberg, founder of the Boston-based firm.

A search for information about drug company Merck, for example, generates a timeline showing not only recent news on earnings but also when various drug trials registered with the website will end in coming years. Another search revealed when various news outlets predict that Facebook will make its initial public offering.

That is done using a constantly updated index of what Ahlberg calls “streaming data,” including news articles, filings with government regulators, Twitter updates, and transcripts from earnings calls or political and economic speeches. Recorded Future uses linguistic algorithms to identify specific types of events, such as product releases, mergers, or natural disasters, the date when those events will happen, and related entities such as people, companies, and countries. The tool can also track the sentiment of news coverage about companies, classifying it as either good or bad.

Recorded Future’s customer base is currently “sub-100,” says Ahlberg. It includes a mix of financial firms, government analysts, and media analysts, who pay a monthly fee to access the online tools. “Government analysts are interested in tracking people and places, while financial services may want to reveal events coming up around particular companies,” says Ahlberg.

As well as providing a slick online interface to perform searches that spit out timelines showing the results (see video), Recorded Future offers free e-mail newsletters that tip users off to predictions in specific areas. It also makes it possible for customers to write software that draws on the tool’s data and analysis through application programming interfaces, or APIS.

In time, this may lead to the development of apps targeted at consumers, says Ahlberg. “If I’m about to buy an iPhone, I might want to know if I am going to look stupid because they’ll launch a new one next week, or how long it usually takes for competitors to launch competing products after a new Apple launch.” Financial analysts are already using the company’s APIs to overlay or even integrate Recorded Future’s data into their own models, he says.

“We have proven out that our data can make strong predictions,” says Ahlberg, citing studies that compared Recorded Future’s output with changes in the volume of activity around particular financial stocks. “We found that our momentum metric, which indicates the strength of activity around an event or entity, and our future events correlate with the volume of market activity,” says Ahlberg.

His company’s tools can also be used to work out which sources of information give the best clues as to future events. A recent analysis showed that the posts on one of the Financial Times’s blogs were better than other news sources at predicting the performance of companies on the S&P 500 share index. Negative posts about a company correlated with below-market performance a week later, while positive ones correlated with above-market performance.

“What they’re really doing here is identifying and collating statements that have been made about the future,” says Steven Skiena at the State University of New York at Stony Brook. Skiena developed similar technology used by another startup, General Sentiment, to mine material from news and blogs. “An analyst can use those to inform their own predictions, less risky than Recorded Future actually making predictions themselves.”

Various tools are capable of extracting events, people, and companies from text, but aligning that information in time is a trickier task, says Panagiotis Ipeirotis, at New York University’s Leonard Stern School of Business. Ipeirotis researches how economically important data can be mined from online news sources and social media. “Analysis of sequences of events is very interesting, and underexploited in the research literature,” he says. “Even getting decently timed data of news articles in order to properly generate event sequences is a hard problem.”

This focus on the timeline sets Recorded Future apart from other firms trying to gain insights by mining news and other data, says Ipeirotis. “I’m curious to see when other text analytics firms will jump into the trend.”

Recorded Future is about to expand its service to cover Arabic and Chinese sources. Making its indexes bigger is a major priority. “I’d like to be able to get in front of every piece of streaming data on the planet,” says Ahlberg.

As the databases covered by Recorded Future, General Sentiment, and others grow, more powerful types of analysis will become possible, says Skiena. “I’m currently working with social scientists on models to predict what the probability is that a person that gets few mentions today suddenly becomes very famous in the future, by looking back at years of past data,” he says.




  1. I would respectfully recommend adding to your list to the right the following (in which I hold no financial interest):
    The Scientific and Medical Network (Google )
    Horizon Research Foundation

  2. Great suggestions, John. Thank you! I just finished adding information about the Horizon Research Foundation… I’ll check out your other suggestions shortly.

  3. An fascinating dialogue is value comment. I think that you need to write extra on this subject, it won’t be a taboo topic however typically individuals are not enough to speak on such topics. To the next. Cheers

  4. Thank you for sharing this article, İt is really Helpful, Thanx/3/26/2011 4:40:30 PM

  5. Really useful. Thanks for your efforts. Appriciated.

  6. Forex

    I have been surfing on-line more than 3 hours lately, yet I never found any attention-grabbing article like yours. It is lovely value sufficient for me. In my opinion, if all site owners and bloggers made excellent content material as you probably did, the web will be a lot more useful than ever before.

  7. Hey :) Is it okay that I kinda off topic? I am trying to read your post on my new Mac but it doesn’t show properly, any suggestions? Should I try to find an upgrade for my software or something? Thanks! Alethia x :)

  8. We would like to thank you once again for the gorgeous ideas you offered Jesse when preparing a post-graduate research plus, most importantly, pertaining to providing the many ideas within a blog post. Provided we had known of your blog a year ago, we may have been kept from the unnecessary measures we were employing. Thank you very much.

  9. I’m really impressed with your writing skills and also with the layout on your blogIs this a paid theme or did you modify it yourself? Anyway keep up the nice quality writing, it is rare to see a great blog like this one these days..

  10. Hi, Neat postThere is a problem with your website in internet explorer, would test this� IE still is the market leader and a huge portion of people will miss your great writing due to this problem.

  11. This is very interesting, You’re a very skilled bloggerI have joined your feed and look forward to seeking more of your fantastic postAlso, I’ve shared your site in my social networks!

  12. wow!! these types of application by chance repeatedly interesting.

  13. Hmm it seems like your post ate my first comment (it was extremely long) so I guess I’ll just sum it up what I wrote and say, I’m thoroughly enjoying) your blog. I too am an aspiring blog blogger but I’m still new to everything. Do you have any tips for rookie blog writers? I’d really appreciate it.

  14. Digital Life, here are a few tips I would offer:

    1. Blog about what really interests you — not what you THINK others may be interested in;

    2. Be sure you get your facts right;

    3. Whenever possible, Include multiple references;

    4. If you publish something and later discover that it is wholly or partially inaccurate, followup and say so.

  15. This formal article helped me a lot! Bookmarked the website. Outstanding information once again! Thanks a lot!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.