Web Crawlers

Information wants to be free. And yet, despite the Web being the largest repository of information in human history, only a tiny fraction of that data is machine readable. This makes the practice of gathering and analyzing data a laborious manual process.

Kimono Labs set out to solve this problem, building an automated data extraction tool that allows even non-programmers to effectively create APIs to collect data from any public website. The product has been a major hit leading to more than tens of thousands of users — it cited 20,000 in March — creating more than 65,000 APIs since the company’s January launch, with each figure growing at a rate of more than 15 percent per week, according to founders Pratap Ranade and Ryan Rowe.

The company lays out its vision to create APIs for the entire Web, structuring the Web and creating the data backbone to power a world of connected devices (IoT) and apps. In the process, the company is making tools like Web crawlers, which were previously available only to programmers, accessible to the rest of society, including academic researchers and journalists, enabling them to conduct large scale analyses of live data.

“There are subject matter experts all around us that know what data sources to trust and what problems or trends are interesting to track, but have no idea how to spin up a Web server or write a program to collect and query this data,” Ranade says. “Before Kimono, these people needed to find a programmer to help them extract this data, and every time they were reinventing the wheel writing these mini-Web crawlers that already existed somewhere. But we’ve tried to make it easy enough that domain experts in all areas can ask any question of the data they want without requiring any technical skills.”

Kimono’s founders measure growth not only in terms of total users and APIs created, but recurring interactions with the product. This is important, because the nature of APIs is that they can be setup and forgotten, but continue to run on autopilot indefinitely. If users doesn’t open and interact with the resulting data, the company doesn’t count that activity in its growth.

The Y Combinator alumnus’s early success came with a downside, which is that despite hosting the world’s largest collection of publicly accessible APIs and, as a result, one of the largest structured information databases around, this information was still a challenge to navigate. Consequently, users have been forced to recreate existing APIs and data queries simply because they didn’t know or couldn’t find prior work that existed within the Kimono community.

Today, the company has removed this barrier with the launch of search. Users can filter existing APIs by name, category, source, and other parameters to find results relevant to their unique needs. Furthermore, every time a user launches the Kimono chrome extension or bookmarklet while on a target webpage, the product will show existing API extraction rules, so users can apply existing API templates to grab relevant data from a website.

The company offers the following example:

With this suite of new features you could search ‘.gov’ to see all APIs created off of a US and UK government domain, discover an existing API extracting, for example, the US employment rate, create a similar API extracting France’s employment rate from another site, combine the APIs into a meta API and comment on the initial API to let users know you’ve created a comparison API to pull US & French data.

Other examples of real APIs created within the Kimono community include a Pebble smartwatch app built off of a Kimono World Cup API, an election app for the recent Indian election, a Bloomberg-like bitcoin correlator, and an API tracking the performance of Nicolas Cage films.

With search, Kimono users can also find existing data sets that may predate their ability to query an existing website. This is important, because one limitation of the Kimono platform is that it can only collect live data and cannot go back in time to collect data that was previously displayed on a website.

Currently, all APIs created on Kimono are public by default, meaning that other users can discover and repurpose these micro-programs. It’s a decision driven by the founders’ belief in initiatives like the Open Data Project. The company, however, plans to roll out a pro feature in the future that will allow users to pay to keep their APIs and the resulting data private. Other potential long-term monetization opportunity that is under consideration is the creation of marketplace for APIs, structured data, and data manipulation tools.

The story behind Kimono’s creation is almost as interesting as the product itself. Ranade and Rowe met during Ph.D. programs in applied mathematics and applied physics at Columbia University in New York. The pair parted ways after grad school with Rowe joining Frog Design as a technologist and creative director and Ranade joining McKinsey & Company as an associate partner focused on strategy in the technology sector. The pair brought the band back together in early 2013, quitting their jobs and traveling through Asia while testing out product ideas.

The initial problem Kimono was created to solve was a personal one. Tired of boarding flights unsure of what the entertainment options would be, the pair decided to create a personal API that would scrape airline websites and organize data around available entertainment for every flight. But when the project proved far more difficult than they expected, the idea for productizing Web scraping was born. The first lines of code for Kimono were written in a cafe in Vietnam called the Hanoi Social Club.

After applying and being accepted to Y Combinator’s Winter 2014 application, the pair moved to the Bay Area and have since grown their team to nine full-time employees. Kimono raised a seed round of undisclosed size and investors following Y Combinator’s recent Demo Day, Ranade says.

With a computer in nearly every pocket today, and with nearly every device around us soon to be smart and connected, there’s more opportunity than ever to make use of the endless sums of data that exist in the world. But, with only 0.0005 percent of the Web accessible through open APIs, according to Kimono estimates, there is a wealth of data that remains unavailable to subject matter experts and decision makers. Kimono is quickly changing this reality and today, with the launch of search, has made the work of its community potentially even more impactful and far reaching.

  1. Kimono Labs allows anyone to turn any website into a structured API in seconds. Kimono is making it easy to find data by eliminating the need to code or install any software for extraction.