The essential idea of SPARKD is to increase inputs from the social environment into newsrooms or any other nexus that seeks to sort, analyse and synthesise raw information into news. Our conviction is that, as an element of society, the Fourth Estate will benefit from radical collaboration outside the newsroom per se, by incorporating (and supporting) these news gathering processes and recognising that there are many points in the chain from capture to publication in which value is being and increasingly will be added.

On this line of thinking, the SPARKD service is premised on five key elements, each of which is designed to be run as a distributed service with a Web frontend

1. Launch – an uploader.
2. Hatch – a transcoder & hasher (hashes identify pieces of news information within the ecosystem)
3. Wire – a nice indexing system
4. BirdSeed – a seeder, allowing anyone to contribute their bandwidth to the greater good.
5. Swift – specifically the developmental Swift player, which allows P2P-powered, web-embedded HTML5 video.

The system is entirely atomic. Any one can run any part of the infrastructure, anywhere. Uploading is handled via HTML5 / HTTP upload and can happen from a mobile device, laptop or desktop, within a widget or through a web browser: it’s just an extension, or a small javascript widget you can add to a blog, post to a social network. KISS. Anywhere can become a point of upload. Keep costs distributed and lower the chance of outright failure.

Processing uploaded files is onerous. At VODO we always have a subsantial queue of broken transcodes and offloading those onto a commercial service is very expensive. It led me to thinking: why not let people run their own transcoding services on their own computers/servers? It’s pretty simple: transcode the file, create  a hash, and add the hash to an index  (as specified by the uploader.) If one transcoder breaks we could concievably shunt the file over to another node to try transcoding it there… The advantages are obvious.

The wire system is an attempt to use PubSubHubBub to create a really useful distributed index. It’s an open protocol for distributed publish/subscribe communication which extends Atom and RSS to allow near-instant, pushed notifications of change updates.  What we’re using it for here is to allow users to maintain indexes which can update and be updated by other indexes according to various permissions and privacy levels.

You could bring all these components together into a single site, which is essentially what Spark’d is – moderated indexes with a specific idea of what constitutes news, with guaranteed privacy and a guaranteed service-level. But the community can also break these components out for their own uses and users — not just to support Spark’d, but to reconsider the very idea of a news/information architecture.

Wikileaks has shown us how important robust, anonymous news infrastructures can be in terms of providing information inputs for comminuties and professionals to analyse and work with, but also how important it is not to leave them in the hands of a particular entity. Absolute power corrupts, and so on… It’s our hope that by open sourcing the components for a distributed information architecture available via any web-enabled device, we’ll provide both increased inputs for collaboration and avoid capture by any single set of agendas.

Note: this post is almost entirely a response to (the last part of) Phillip’s question from last week! Hope to deal with its first part in a future post.

The problem: while news media plays an increasing role in defining and setting political agendas (see Fox News, George Bush, election of; Fox News, Tea Party, creation of;  Every Outlet Ever, Weapons of Mass Destruction in Iraq, Phantom Menace of), its working practices are over-determined by corporate/business agendas – in effect, regulatory capture of the Fourth Estate.*

Anyone in doubt about the exact degree of consolidation (the process by which single companies have come to own the vast majority of the news apparatus) can have a look here and here. Or listen to what Dan Rather has to say:

Dan is right to be concerned. The massive cuts in foreign correspondent posts in Western news media have been noted by Human Rights Watch in “Whose News?”, from which I’ll risk a massive quote:

Paradoxially, it is precisely in wealthier countries where the media are sickest today. In the United States, the triple blow of the internet, the economic recession, and the poor management of a few major newspapers has dramatically shrunk the cadre of foreign correspondents.  Several daily papers, such as the Boston Globe and Newsday, have shut down their foreign bureaus entirely. Television networks have closed almost all of their full-fledged bureaus, leaving local representatives in a handful of capitals. The New York Times and The Washington Post, the reigning monarchs of international coverage, appear to maintain their foreign bureaus more out of the personal commitment of the families who still own them. In the United States, at least, the commercial model for international fact-gathering and distribution is evidently broken.

No one is more vocal about the dire consequences of these cutbacks than the newspapers’ foreign correspondents themselves. Pamela Constable, a respected foreign correspondent for The Washington Post, wrote in 2007: “If newspapers stop covering the world, I fear we will end up with a microscopic elite reading Foreign Affairs and a numbed nation watching terrorist bombings flash briefly among a barrage of commentary, crawls, and celebrity gossip.

I offer this a serious problem, one that we ought to have the courage to address.  A solution needs to counter the wholesale stripping down of news gathering:

• Provide increased inputs from the ground, taking into account the lack of access, security and bandwidth at likely input points. Ideally there should be incentives for inputs into the system, but these could be exogenous.
• Abstract raw ‘reporting’ from analysis, so that filtering and expertise are applied at a later stage; recognise that while the expert analysis is needed (cf. catastrophic results in analysis pre-Iraq, when Rumsfeld cut out the CIA and started to analyse his own raw intelligence data, getting his own results; cf. “bloggers just repeat what the mainstream news says”) it may not be the same kind of analysis applied currently by the news media (e.g., the rise of long-form reading w/r ebook readers, tablets etc.)

• Recognise that no one system is sufficient to introduce end-intelligence into the data: design to produce atomic chunks of news, allowing others to apply metadata, their own analyses, their own clustering. Create a referencing system for each new chunk – link these within stories. A chunk may not make sense until it finds its story, even years later. But every chunk has potential value.

•  Make the system redundant and archival; distribute it so that it remains free of singular control. Every single centralised system will begin to exhibit particular censorship qualities given enough growth and commercial or state pressure. Design against this from the beginning.

Previous media models were designed around limited bandwidth and high capital investment. It makes no sense to design specifically around TV, radio and  print now, when we have an entirely different set of constraints. To not risk refracting the current problems described above, we have to think down from surface–skeleton to structure–scope–strategy:

Spark’d was intended to react to these problems as much as possible. It emulates the maligned durability of P2P networks in a news infrastructure that stays away from a central portal or site, uing HTML5 to distribute component activities (report, analyse, redistribute, reorganise) wherever they need to take place. I am grateful to anyone who will enagage with and critique the idea. My hope is that if we can address these problems head on, we’ll provide an infrastructure that helps re-imagine the future of news, not just as industry, but as a vital part of our society.