I made this as a Christmas/New Year present for my friends:
SPARKD PROJECT PROPOSAL
DESIGN BRIEF
SUMMARY
SPARKD is intended to increase inputs from the social environment into newsrooms or any other nexus that seeks to sort, analyse and synthesise raw information into news. The service is premised on five key elements that we label, as a whole, “Flit”. Each element is designed to be run as a distributed service with a Web frontend:
- Launch – an uploader.
- Hatch – a transcoder & hasher (hashes identify pieces of news information within the ecosystem)
- Wire – a distributed indexing system
- BirdSeed – a seeder, allowing anyone to contribute their bandwidth to the greater good.
- Swift – specifically the developmental Swift player, which allows P2P-powered, web-embedded HTML5 video.
The system is entirely atomic. Anyone can run any part of the infrastructure, anywhere.
HOW IT WORKS
1. LAUNCH
After a video is captured a user can upload it to any Hatch server to be transcoded and indexed. This can be done using a normal browser page, a Firefox add-on or an app if on a mobile device. Metadata such as author, location and tags can be entered or is collected automatically by the device.
Uploading is handled via HTML5 / HTTP upload and can happen from a mobile device, laptop or desktop, within a widget or through a web browser: it’s just an extension, or a small javascript widget you can add to a blog, post to a social network. Anywhere can become a point of upload. Keep costs distributed and lower the chance of outright failure.
Processing uploaded files is onerous : why not let people run their own transcoding services on their own computers/servers? It’s pretty simple: transcode the file, create a hash, and add the hash to an index (as specified by the uploader.)
2. HATCH
After the file has been uploaded, the Hatch server will analyze the video and transcode it into the necessary formats to be ready for web streaming (WebM, Theora and H.264). A hash is created and the file is submitted to one or several indexes and seeders depending on the information supplied by the user and the Hatch server settings.
Once the transcode is completed, the Hatch server pushes the hash back to the Launch server and the end-user who can then add it to his watchlist inside the Firefox add-on or app.
The Wire collects hashes and additional metadata for video files and provides this information to end-users, seeders and any web service that utilizes the content. A list of videos can be subscribed to by any RSS service or from inside the Flit Firefox add-on.
3. BIRDSEED
Any user or organisation who wants to support a specific wire can download and install Birdseed, which will automatically download and seed all videos added to the Wire.
4. SPARKD — BRINGING IT ALL TOGETHER
While any of the Flit services can be run independently most of the time, a single server can provide all of the Launch, Hatch, Wire and Birdseed functionality to enable a video to be uploaded and seeded immediately. Sparkd is a web service that in addition to the bare-bone indexing and seeding service also provides a web front end and community where users can watch and discuss all the videos submitted to Sparkd. Sparkd will run as a stand-alone web service buy any content can easily be embedded on other websites and established news channels could incorporate a Sparkd channel to complement their reporting on a specific and current topic.
FIREFOX ADD-ON
The Flit Firefox add-on will enable users to upload and index content straight from their web browser, either by using a public Hatch service or by transcoding and submitting their videos straight to a Wire. After a video has been submitted they can keep track on the status of the video and see how many people are streaming and sharing it.
Core benefits of this approach:
- Cost of distribution is offloaded onto the audience itself. – No central capital expenditure makes censorship and other such problems less likely.
- Atomised infrastructure allows value to be added “down the chain” in unpredictable ways. SPARKD is designed as a ‘minimum viable product’, facilitating communities in distributing media through their social graph. Both the original publiser and consequent viewers are able to mark up SPARKD videos in such a way as to create an emergent narrative trail between videos, which is readable both from within the SPARKD/flit ecosystem and within social network such as Twitter and Identica utilising the hashtag system. This allows people to identify videos in which they’re interested, but also can pull conversation, for example from the Twittersphere, into the HTML5/video playback environment.
- SPARKD videos are watched wherever they are embedded, and constantly recontextualised with this extrinsic social data, according to display motifs to be explored here. Urgent visual information can be drawn into ongoing conversation, displayed via or within social media clients, or responded to from tablet or phone devices.
USE-CASE
A citizen in Libya captures a sensitive video on her phone which she wishes to share. She selects the level of anonymity and privacy she needs and uses the free flit app for her Android phone to upload the information anonymously to the SPARKD server. The video is transcoded, indexed and made available for streaming.
As events develop in Libya, those monitoring SPARKD, its RSS or social media outputs have seen the ‘Libya’ index appear. They now note a new anonymous video has appeared, which they are immediately able to watch as it is seeded by SPARKD. As they watch, they too become seeders of the video, and as they begin to spread it through their social graphs, so to do their connections.
Some users choose to start seeding the entire Libya index, while others support only this particular video. They can spread the film through their social networks via a shortlink, or embed it as they might a YouTube video.
The video turns out to have political importance and a State security service objects. SPARKD is compromised either by legal attacks or hacking and its entry is removed from the maintained index. However, because knowledge of the hash is all that is needed to extract the video from the P2P swarm, the anonymous video is still able to be accessed. Moreover, the indexing system used by SPARKD means that the index itself has already been replicated and recombined many times, making the attack insufficient to prevent access to the video.
BUSINESS CASE
The problems Sparkd solves: insufficient inputs from the grassroots / location / street-level; insufficient opportunities for users to sort this data before it hits the newsroom’
How Sparkd facilitates telling a story: with information moving so fast, often outpacing the newsroom’s capacity to keep up, it’s important to facilitate increased speed of inputs into the newsroom. Stories should be directly connected to immediate experiences and wherever possible to citizen media. The disconnect between media and citizen should be broken down. The organisation that does this will become full of win.
Who will benefit: citizen journalists will feel increasingly connected to the news gathering process, seeing their work filter into network news. Journalists will benefit by being exposed to the real concerns of local people even if those concerns are not ready for primetime.
Does your product leverage off other tools? Yes, this project is premised on exploring the groundbreaking Swift embedded, P2P streaming video protocol in conjunction with P2P Next, a partnership we’ve already established.
Why does Sparkd make sense to build from a news organization’s perspective? A newsroom needs to connect with the populace. Sparkd gives it means to do that. Being at the center of an infrastructure that improves people’s capacity to communicate with one another audiovisually, without trying to dominate or own that infrastructure, will be of immense benefit to positioning an organisation for the future of news gathering.
Let A Thousand Newsrooms Bloom
The essential idea of SPARKD is to increase inputs from the social environment into newsrooms or any other nexus that seeks to sort, analyse and synthesise raw information into news. Our conviction is that, as an element of society, the Fourth Estate will benefit from radical collaboration outside the newsroom per se, by incorporating (and supporting) these news gathering processes and recognising that there are many points in the chain from capture to publication in which value is being and increasingly will be added.
On this line of thinking, the SPARKD service is premised on five key elements, each of which is designed to be run as a distributed service with a Web frontend
- Launch – an uploader.
- Hatch – a transcoder & hasher (hashes identify pieces of news information within the ecosystem)
- Wire – a nice indexing system
- BirdSeed – a seeder, allowing anyone to contribute their bandwidth to the greater good.
- Swift – specifically the developmental Swift player, which allows P2P-powered, web-embedded HTML5 video.
The system is entirely atomic. Any one can run any part of the infrastructure, anywhere. Uploading is handled via HTML5 / HTTP upload and can happen from a mobile device, laptop or desktop, within a widget or through a web browser: it’s just an extension, or a small javascript widget you can add to a blog, post to a social network. KISS. Anywhere can become a point of upload. Keep costs distributed and lower the chance of outright failure.
Processing uploaded files is onerous. At VODO we always have a subsantial queue of broken transcodes and offloading those onto a commercial service is very expensive. It led me to thinking: why not let people run their own transcoding services on their own computers/servers? It’s pretty simple: transcode the file, create a hash, and add the hash to an index (as specified by the uploader.) If one transcoder breaks we could concievably shunt the file over to another node to try transcoding it there… The advantages are obvious.
The wire system is an attempt to use PubSubHubBub to create a really useful distributed index. It’s an open protocol for distributed publish/subscribe communication which extends Atom and RSS to allow near-instant, pushed notifications of change updates. What we’re using it for here is to allow users to maintain indexes which can update and be updated by other indexes according to various permissions and privacy levels.
You could bring all these components together into a single site, which is essentially what Spark’d is – moderated indexes with a specific idea of what constitutes news, with guaranteed privacy and a guaranteed service-level. But the community can also break these components out for their own uses and users — not just to support Spark’d, but to reconsider the very idea of a news/information architecture.
Wikileaks has shown us how important robust, anonymous news infrastructures can be in terms of providing information inputs for comminuties and professionals to analyse and work with, but also how important it is not to leave them in the hands of a particular entity. Absolute power corrupts, and so on… It’s our hope that by open sourcing the components for a distributed information architecture available via any web-enabled device, we’ll provide both increased inputs for collaboration and avoid capture by any single set of agendas.
Let’s Talk About The Problem (MoJo Learning Lab, Week #2)
Note: this post is almost entirely a response to (the last part of) Phillip’s question from last week! Hope to deal with its first part in a future post.
The problem: while news media plays an increasing role in defining and setting political agendas (see Fox News, George Bush, election of; Fox News, Tea Party, creation of; Every Outlet Ever, Weapons of Mass Destruction in Iraq, Phantom Menace of), its working practices are over-determined by corporate/business agendas – in effect, regulatory capture of the Fourth Estate.*
Anyone in doubt about the exact degree of consolidation (the process by which single companies have come to own the vast majority of the news apparatus) can have a look here and here. Or listen to what Dan Rather has to say:
Dan is right to be concerned. The massive cuts in foreign correspondent posts in Western news media have been noted by Human Rights Watch in “Whose News?”, from which I’ll risk a massive quote:
Paradoxially, it is precisely in wealthier countries where the media are sickest today. In the United States, the triple blow of the internet, the economic recession, and the poor management of a few major newspapers has dramatically shrunk the cadre of foreign correspondents. Several daily papers, such as the Boston Globe and Newsday, have shut down their foreign bureaus entirely. Television networks have closed almost all of their full-fledged bureaus, leaving local representatives in a handful of capitals. The New York Times and The Washington Post, the reigning monarchs of international coverage, appear to maintain their foreign bureaus more out of the personal commitment of the families who still own them. In the United States, at least, the commercial model for international fact-gathering and distribution is evidently broken.
No one is more vocal about the dire consequences of these cutbacks than the newspapers’ foreign correspondents themselves. Pamela Constable, a respected foreign correspondent for The Washington Post, wrote in 2007: “If newspapers stop covering the world, I fear we will end up with a microscopic elite reading Foreign Affairs and a numbed nation watching terrorist bombings flash briefly among a barrage of commentary, crawls, and celebrity gossip.
I offer this a serious problem, one that we ought to have the courage to address. A solution needs to counter the wholesale stripping down of news gathering:
- Provide increased inputs from the ground, taking into account the lack of access, security and bandwidth at likely input points. Ideally there should be incentives for inputs into the system, but these could be exogenous.
- Abstract raw ‘reporting’ from analysis, so that filtering and expertise are applied at a later stage; recognise that while the expert analysis is needed (cf. catastrophic results in analysis pre-Iraq, when Rumsfeld cut out the CIA and started to analyse his own raw intelligence data, getting his own results; cf. “bloggers just repeat what the mainstream news says”) it may not be the same kind of analysis applied currently by the news media (e.g., the rise of long-form reading w/r ebook readers, tablets etc.)
- Recognise that no one system is sufficient to introduce end-intelligence into the data: design to produce atomic chunks of news, allowing others to apply metadata, their own analyses, their own clustering. Create a referencing system for each new chunk – link these within stories. A chunk may not make sense until it finds its story, even years later. But every chunk has potential value.
- Make the system redundant and archival; distribute it so that it remains free of singular control. Every single centralised system will begin to exhibit particular censorship qualities given enough growth and commercial or state pressure. Design against this from the beginning.
Previous media models were designed around limited bandwidth and high capital investment. It makes no sense to design specifically around TV, radio and print now, when we have an entirely different set of constraints. To not risk refracting the current problems described above, we have to think down from surface–skeleton to structure–scope–strategy:
Slide from Jesse James Garrett, from his lecture on user experience
Spark’d was intended to react to these problems as much as possible. It emulates the maligned durability of P2P networks in a news infrastructure that stays away from a central portal or site, uing HTML5 to distribute component activities (report, analyse, redistribute, reorganise) wherever they need to take place. I am grateful to anyone who will enagage with and critique the idea. My hope is that if we can address these problems head on, we’ll provide an infrastructure that helps re-imagine the future of news, not just as industry, but as a vital part of our society.
VODO Is Netflix-Meets-Kickstarter for Indie Film Fans | Fast Company
Notes on Week #1 of the Knight/Mozilla Learning Lab
In the spirit of being: an ingrate, an awesome-resistant European and just plain contrary, what follows are my hopefully not un-constructive, or at least not merely rude, comments (they made me do it! It’s being marked! I even have a word limit!) on the last week of MOJO lectures.
#1 Please Could The Medium Stop Being The Message Now ?
Mr. Raskin. I am concerned that striving from the first for a better presentation of one’s idea (in order to get people agree on its value, and, presumably, resource it) skews it towards a path of least resistance / greatest acceptance which might very well warp its very essence. If you will allow me the idea of a fickle zone in which good looks and ‘polish’ are all too readily rewarded, then you may imagine the potential to ‘iterate’ not towards ‘solution’, but towards triviality. In which kind of world, I want to ask, would every solution immediately be presentable as a UI?
#2 Resist Aimless Hypervelocity!
‘Aim to finish the first–pass in a day’. It’s at least possible that applying said heuristic (don’t you love the word ‘heuristic!’ it’s like ‘iterate’! It makes us sound like scientists!) has as much chance of collapsing a vision than expanding it. (But never mind! It appeals to the breakneck velocity with which everything needs to be executed today!) Would it be utterly contrary to suggest that most valuable ideas might precisely be those that no-fucking-way could-be-prototyped-in-a-day?
(Having said that, I love this.)
#3 An ill-favoured thing Sir… but mine own.
So then, if you can’t communicate it – to an investor, your company, MOJO – never mind, find a way! Execute it with the only lunatics in the world that’ll work with you. Make it ugly! They are often wrong and that is the whole point of them! While they iterate heuristics, you will have made your imperfect thing. May it prosper.
#4 Iterate this, Buster?
Here (from 1 minute in) journalist Nick Davies, author of Flat Earth News, adroitly describes the problems with the entire enterprise of news today, and how we (that is developers of technology) are implicated in them:
Problems arising from our consolidated , streamlined, technologised ‘news’ machine:
- Most news these days comes from news agencies (Reuters, Press Association etc);
- Of the remainder, a very high proportion comes from PR sources (corporate or governmental), with a high degree of inherent bias;
- Perhaps as little as 12% of the news we read is actually generated by reporters.
- News websites run by media firms recycle 50% of their stories from the two international wire agencies, Associated Press and Reuters; those run by internet firms recycle 85% of their stories from those two.
- Compounding the problem the wires are reducing their stringers, getting their material from PR’s too! PR people now outnumber journalists.
When we think about solving a problem of this magnitude (‘news’ just doesn’t exist in the way we might imagine!), I would argue that we need to think deeply, slowly and towards a goal that can benefit the entire ecosystem. Recognising the critical nature of the infrastructure we are aiming to operate on, it seems to me okay if, at the beginning at least, we are not immediately polished, slick and presentable. It would be okay, actually, to be downright unpopular!
(British playwright) Dennis Potter is instructive, just before his death and prescient on the topic of Rupert Murdoch! Awesome!
Egyptian State Security Archives Yield Evidence of False Flag Attacks: Reason Magazine
Evidence emerging of further false-flag operations to sustain the War on Terror in Egypt, including the Sharmh-el-Sheik bombings:
Up until now, claims of terrorism have been the most effective way for Arab dictators to get sympathy and support from the US. (The Yemeni regime, which is now teetering on the edge of collapse, saw its aid double after the Christmas 2009 attempted underwear bombing.) American policy in the region has been predicated on the Faustian bargain that we overlook Arab dictators’ shoddy human rights record and continue to prop them up in exchange for stability and a hard line on Islamic terrorism. But the Egyptian State Security archives suggest that not only were the Mubaraks not delivering an end to Islamic radicalism, but the regime itself may have been the source of much of Egypt’s terrorism and sectarian strife.
Egyptian State Security Archives Yield Evidence of False Flag Attacks – Hit & Run : Reason Magazine.
When capitalists begin to speak of capitalism….
… recently has Marxism been back on the agenda, placed there, ironically enough, by an ailing capitalism. ‘Capitalism in Convulsion’, a Financial Times headline read in 2008. When capitalists begin to speak of capitalism, you know the system is in dire trouble.
from Terry Eagleton’s review of How To Change The World: Marx and Marxism, by Eric Hobsbawm in London Review of Books.
The REAL Death Of The Music Industry
The REAL Death Of The Music Industry. Now these are some instructive graphs.
The content singularity is nigh.
Blogged From My Walking Desk
By now, quite few people are aware that I have converted from my usual practice of slouching around on a sofa / lying in bed / sitting at a desk / standing at the kitchen worksurface while working to walking on my ‘walking desk‘.
Simply put, the walking desk is a standard treadmill (of the type you see in gyms or standing unused in bedrooms and garages) with two tables over it, one for the keyboard (in my case a whole MacBook Air) and one for the monitor. The keyboard just above waist height (assuming you still have a waist by now) and the monitor at around head height (actually the top of the monitor is about at eye level.)
I built the desk myself from bits of used wood I retrieved from the Bristol Wood Recycling Project. I had no experience at all working with wood or building anything, really, so I imagine this would be pretty easy for anyone who does, or indeed anyone at all, since I’m peculiarly inept and particularly left-handed in such matters. My desk is fairly stable and does the job.
It’s been three weeks now since I built the desk, and other than 5 days away at IFFR Labs, I have used the desk every day. I now walk at about 1.4 MPH at a resting speed, and about 2.1 MPH at a maximum during the day. This means I am now walking between 7 and 10 miles a day, depending on how long I spend in front of the computer. (I’ve also been trying to change screens for the Kindle to read long articles and books at least a couple of hours a day.) I can say without any reservation that the walking desk is a perfectly practical way to work and I am able to carry out all essential tasks with no noticeable degradation of quality or speed.
The logic behind this radical change of working environment is pretty simple. It’s become obvious to me that I’m going to spent the majority of my life working via computers and this seems a good way of dealing with the problem of spending the majority of my day physically inactive while doing so. It seems to me that incorporating lots of walking into daily life is more sensible that trying to cram it all in to a gym session once a day (like anyone sane makes it to the gym once a day). Even if you do make it to the gym for an hour or so, you’re fighting a losing battle against the other 6-7 hours you spent immobile.
This allows me to indulge my usual behaviour of starting working as soon as I get up without guilt. I drink tea at the walking desk. I (sometimes) eat at the walking desk. I watch films at the walking desk. And when I’m done at the end of the day I am now physically tired, which, frankly, is great. It’s possible that the walking desk is the path to ultimate salvation, but then I haven’t tried Scientology.
Other than the obvious effect of gradually losing weight (though I am not measuring this in any objective fashion), I have noticed one interesting consequence of using the walking desk. You really begin to quantify time spent in front of the computer differently — both noticing how much time the internet ‘takes’ from you in its many diversions, and feeling that (since you are walking) you should maybe use that time more productively. There is a way that the machine has of ‘parcelling out’ time that stops you from floating off into an ‘oh, did four hours just pass? I really have been doing useful stuff…’ state of mind. It’s as if this is just a lot less possible when you’re walking at 2mph.
I am going to post some pictures of my walking desk soon. Here, for what it’s worth, are some guidelines I observed making the thing:
1) Of course you can and should use a second hand treadmill. I imagine the amount of energy that goes into making these things is a horror compared to the amount of use they get before being trashed. The people I bought mine from had a great laugh when they found out what I was doing with it. It cost me £100, though I could probably have got it for £80 if I was being a tightwad.
1b) And yes, you need a powered treadmill for this.
2) Make sure the treadmill you buy goes slow enough. You’re probably going to end up walking between 1.2-2.5 MPH, so you could check those kinds of speeds.
3) Check how noisy the treadmill is. Mine is quite loud. I replace it with something quieter at some point, but the manufacturers don’t post the decibels of their products, so this is going to take a bit of time.
4) I don’t suggest getting into any complicated structure-planning until you actually got your treadmill in front of you. They have bits of moulded plastic poking out here and there that could well affect your plans.
5) It’s an ideal thing to build out of bits of reclaimed wood. I used the really cool Bristol Wood Recycling Project. They gave me some useful advice on building and it cost me about £20.
6) I strongly suggest you build the desks so that they stand free above, over or around the treadmill. The treadmill will move. Personally I don’t want my desk shuddering around, and that’s what happens if you just lay a plank across it. Believe me, I tried it.
7) Make the monitor-bearing desk high enough that the top of your monitor can be about level with your eyes.
Make the computer or keyboard-bearing desk high enough that your arms rest on it comfortably.
I can’t think of anything else. If you work at home or in an office that would accept this kind of lunacy, I suggest you get to it now.