Yahoo! Pipes – the Edwin Pipe in under 15 minutes
At the MySQL Conference the closing keynote was on Yahoo! Pipes, by Pasha Sadri, a Principal Software Engineer, Advanced Development Division, Yahoo!. I wanted to try it, but I was on Firefox 1.5 on Fedora Core 6 and there was no way I was going to build a pipe during the talk.
Fast forward a week or so later, and a boring Friday night ensued. What better thing to do, than to play with Pipes. In under fifteen minutes, I created the Edwin Pipe. What is it? Its a pipe that is all things MySQL – comprehensive source of news, whats cool, and so forth. There are some limitations – regular expression support is supposedly like Perl’s, but is not quite complete. The Unique operator is pretty cool, filtering is good (can be improved with better regex support), and maybe some sort of fuzzyness in the way data is displayed (I don’t only want all Digg mysql related items popping up at the top, or I don’t only want all mysql job forum details at the bottom, etc.). Language conversion via a Babelfish operator exists, but not language filtering (maybe I only want all English text displayed in my final pipe output).
That aside, the forums are pretty active. Pipes are ridiculously easy to create. Its simply great stuff. Oh, shorter URLs – the URLs are so long and not feasible, in my opinion. Impressive is the support to then get RSS output, and also JSON (so all processing is done on the server side). Happy I am with sites that provide JSON feeds.
Now, for some notes I took during the closing keynote.
- A while ago, he wanted to find an apartment near a park. Go to Craigslist and find apartment lists, then click the map link, and also check distance to a park on the map… This is tedious, and not automated.
- Craigslist apartment RSS feed. Yahoo! Local API to find Parks. Why not tie this in together? It started with about 50 lines of Perl code, and it combined feeds + web services (this is your Web 2.0 mashup).
- Pipes: free online service that lets you remix data and create mashups using a visual editor.
- Pipes treats the web as a big database, as they do joins across different ‘tables’.
- Design principles came from the Unix Pipes. They’re like pipes for the Web. Build useful applications from simple primitives.
- The more open Pipes is, the more useful it will be (so Google goodness will also work).
- Output available in JSON, so it can be used as another application. Get email or SMS from output, even. RSS is obviously available.
- App Examples: Last.fm + Flickr, Babbler (Second Life, language translation) by Max Case.
- Must enable users to solve ad-hoc problems. User generated “features” and disposable applications -> the future.
- Pipes uses MySQL, squid for caching, PHP & Perl (lots of CPAN modules) for serving and back-end processing of the pipes.
Edwin 2.0 is already in the works. It will have more cool feeds, and probably work out all the language issues with more separated regexes. More fuzzy organizing of data, if possible. If you want to see a MySQL Blogger Photo Gallery, bman_seattle created a pipe too.
Technorati Tags: mysqlconf, mysqlconf07, yahoo, pipes, edwin pipe, json, visual editing, mashup
