The essential idea of SPARKD is to increase inputs from the social environment into newsrooms or any other nexus that seeks to sort, analyse and synthesise raw information into news. Our conviction is that, as an element of society, the Fourth Estate will benefit from radical collaboration outside the newsroom per se, by incorporating (and supporting) these news gathering processes and recognising that there are many points in the chain from capture to publication in which value is being and increasingly will be added.
On this line of thinking, the SPARKD service is premised on five key elements, each of which is designed to be run as a distributed service with a Web frontend
- Launch – an uploader.
- Hatch – a transcoder & hasher (hashes identify pieces of news information within the ecosystem)
- Wire – a nice indexing system
- BirdSeed – a seeder, allowing anyone to contribute their bandwidth to the greater good.
- Swift – specifically the developmental Swift player, which allows P2P-powered, web-embedded HTML5 video.
The system is entirely atomic. Any one can run any part of the infrastructure, anywhere. Uploading is handled via HTML5 / HTTP upload and can happen from a mobile device, laptop or desktop, within a widget or through a web browser: it’s just an extension, or a small javascript widget you can add to a blog, post to a social network. KISS. Anywhere can become a point of upload. Keep costs distributed and lower the chance of outright failure.
Processing uploaded files is onerous. At VODO we always have a subsantial queue of broken transcodes and offloading those onto a commercial service is very expensive. It led me to thinking: why not let people run their own transcoding services on their own computers/servers? It’s pretty simple: transcode the file, create a hash, and add the hash to an index (as specified by the uploader.) If one transcoder breaks we could concievably shunt the file over to another node to try transcoding it there… The advantages are obvious.
The wire system is an attempt to use PubSubHubBub to create a really useful distributed index. It’s an open protocol for distributed publish/subscribe communication which extends Atom and RSS to allow near-instant, pushed notifications of change updates. What we’re using it for here is to allow users to maintain indexes which can update and be updated by other indexes according to various permissions and privacy levels.
You could bring all these components together into a single site, which is essentially what Spark’d is – moderated indexes with a specific idea of what constitutes news, with guaranteed privacy and a guaranteed service-level. But the community can also break these components out for their own uses and users — not just to support Spark’d, but to reconsider the very idea of a news/information architecture.
Wikileaks has shown us how important robust, anonymous news infrastructures can be in terms of providing information inputs for comminuties and professionals to analyse and work with, but also how important it is not to leave them in the hands of a particular entity. Absolute power corrupts, and so on… It’s our hope that by open sourcing the components for a distributed information architecture available via any web-enabled device, we’ll provide both increased inputs for collaboration and avoid capture by any single set of agendas.