I saw the various bits of web spoo about Yahoo Pipes this past week, and the first thing that came to mind was “Wow – it looks like just Quartz Composer!”.
I guess that’s a good thing – the visual data-flow style “language” (is markup a better term) is fairly easy to follow, even if it takes up a crap-ton of visual space. There’s some good examples out there now, even with the lurch that Yahoo had launching the ‘product’. I’m not even sure that I can really describe it as a product. Although I don’t know what else to call it.
Conceptually, I think it’s brilliant. It’s taking the concept of unix pipes (which I have embraced from years ago in my hacky shell “gettin’ something done” scripting) and applying it to something that you might call reasonably structured data – RSS.
I actually rather expect the structure of RSS to make things more difficult, even while it enables significantly more power to the process as well. With ye olde Unix style pipe, the most powerful tools worked from data streams of plain text delimited with line endings. The mighty grep, find, awk, sed combination. There’s still problems I solve with a habital scattering of those commands between |’s.
RSS (I’m including Atom in this horrific overloading of a specific term) has the benefit of being reasonably consistent in structure. There’s all sorts of nasty additions and complications to it, but you’ve got to admit – it’s a hell of a lot more regularly structured than the results of JDBC, ODBC, or Web (xml trash) Services. It’s simple. It has a few syntactic regular structures – and it can hold a representation of just about anything, albeit some less efficiently than others. Shoot – I’m using it day to day in not only web feeds, but build systems results and dashboards metrics for services that I manage.
“Back in the day” (where the hell did that phrase come from anyway?) I fiddled with a language called Prograph that I desperately wanted to love. It was clunky, and ultimately not fulfilling in it’s efficiency of getting things done – but I wanted a visual mechanism for programming – I dunno, cause I’m a visual kind of guy. Variations of that markup/language live on in the form of Marten and the open source project The Open Prograph Initiative. A neat concept, and I think the precursor to what Yahoo Pipes (or equivilant kinds of things) could be.
And of course I’d be remiss without mentioning the MacOS X platform specific goody that’s more generic – Anthracite – a product that Joe Pezillo has been cobbling on for a good couples of years. It does the RSS routing and twiddling thing too – as well as a whole lot of other data types. And then of course there’s Automator – no forking in it with simple use, but effectively another set of pipes where you stage data flowing from one process to another, hopefully coming out with something useful in the end.
The thing that really brings Pipes to the front is that it’s editable on the web. “Just a matter of time” maybe, before someone did it – made a visual dataflow that you could edit with the javascript/ajaxy goodness that’s been worming back into the foreground again. Still, just having done it is a major step forward.
Beyond all that babbling, I don’t really know where it’s going to go – or how useful it’ll ultimately be to me personally. Of course I’ll fiddling and find out, but at the moment I’m just excited by seeing that it exists.
Rhonabwy 