Experiences in Crowd Sourcing

Mechanical Turk

Daniel Arbix

I have signed up to Mechanical Turk as a Brazilian citizen (not all HIT opportunities were available, then). I have tried to explore different tasks to check the diversity the platform offers for workers. There are some amusing tasks to be performed, but most – against the website’s Participation Terms –, are boring schemes to distort internet advertisement payments or to gather active e-mails (for spam or worse, I presume). There are also numerous HITs which demand fake reviews of products and websites, or which require “turks” to show support to social-networking profiles or events, and even to write posts in blogs making compliments to the blog owner’s clever analyses. A short summary of my experiences follows.

1st try: bad page design, no $ I saw a task as available, was able to follow the instructions and actually perform it on a third party website, but then I realized that the event that would trigger a confirmation number required for my payment had already expired, so of course no payment was made to my account. This was the first contact I had with HITs designed to redirect traffic to earn advertisement revenues (it was described as “a test for page load time, very easy, for USD 2.00”).

2nd try: the second HIT I tried had the following misleading instructions:

• "This HIT is an easy to complete 'sign up' assignment. It shouldn't take you longer than 2 minutes to complete. Many thanks for your time! Simply go to: http://www.awin1.com/awclick.php?mid=633&id=93282 Sign up. Then send a print screen of your confirmation email to: mark.studentearnings@gmail.com and write your username and approximate time of sign up in the box below"

The site redirected me to http://www.offersclick.co.uk/offers/SiteRender.aspx?SiteID=501&ThemeID=21&q14259=AFW&q26274=93282 Again, it seems like arbitrage of internet advertisement revenues… It was impossible to complete the task within the assigned time – the form-filling advertisements website took too much to load –, so I returned the HIT (and received no $, in spite of losing a lot of time)

3rd try: the HIT had the following description:

• "Complete free online quote form. MUST BE 18 OR OLDER AND LIVE IN USA TO COMPLETE THIS HIT Visit Site - Enter Name and Email - Takes less than a minute. IMPORTANT: Eligible only to those who have not yet signed up to this offer. Please don't use disposable e-mails as well. Let's keep this site honest. INSTRUCTION: 1) To get started, visit this website: http://www.aislezone.com/mturk-offer02.php 2) Select make, model and enter zip code. Click "get quotes" 3) Complete the form and click on "get free dealer quotes" REQUIRED PROOF: On activation confirmation copy/paste or type full text to the box below starting with : "Sent!..." "

The time-description was fair, and I received payment after two days. I also (stupidly) provided my real Stanford e-mail, with the result of getting my mailbox now filled by car dealers messages. Again, the HIT seems like arbitrage of internet advertisement revenues.

4th try: the HIT had the following description:

• "Data Collection-Google Results Reward: $1.25 per HIT Insert address, perform google searches and indicate first result"

This HIT was an honest one, requiring a verification of search results (Google), all related to the same company. Time description was fair, and my $ arrived a day later.

5th try: the HIT had the following description:

• "Rate faces of candidates running for office. Usually takes less than 5 minutes. You can find the survey here: http://www.surveygizmo.com/s/207922/faces3 At the end of the survey, you'll find a code. To get paid, please enter the code below:"

This was a fun, quick HIT. It had a fair time description, and payment was received after a day. I was, however, a terrible subject for the MIT social psychology survey, since most pictures to be rated are of Brazilian politicians…

Jason Harrow

It all started out so promisingly. Before I even started, I was excited because "Mechanical Turk" is such a great name. What did it mean? What was mechanical about it? And who or what is the “Turk” in the transaction? Then, in the back of my mind: isn’t that somehow vaguely racist?

Wikipedia would know what a Mechanical Turk is, of course. So I searched for it, and I immediately found my answer. The Mechanical Turk was a chess playing “machine” that, “from 1770 until its destruction by fire in 1854 . . . was exhibited by various owners as an automaton, though it was explained in the early 1820s as an elaborate hoax.” Something thought to be a form of artificial intelligence later explained to be just a smart guy in a box? How cool is that? Could there be a more perfect name for this service? I made a mental note that if I am ever in the position to pick a name for an Internet company or service, I will find out who chose "Mechanical Turk" and consult him or her. I couldn’t have been more excited to get started.

I logged in with my Amazon ID and created my Amazon payment account. Unlike Daniel (above), I am a US citizen; also unlike Daniel, I chose to go first for a little assignment that paid $.02 per HIT. My task was to go to the website of a given educational institution, type in when the spring semester 2010 started, and give the URL of the Academic Calendar page confirming this. I did one successfully and submitted it. It took about 90 seconds. Easiest job of my life. Bring on a few more.

The next one took longer and bore no fruit, though: the little Midwestern barbershop college that was assigned to me didn’t display when their spring semester started (side thought: do barbershop colleges even have spring semesters?). That was frustrating; all that effort, and no shiny pennies at the end of the rainbow. And why in the world does this guy – Aaron Smyth, whoever that is – even want to include barbershop colleges in his survey? What could he possibly be doing with that data? I pressed on and did a few more start dates before trying to find something else. It got boring pretty quickly, and I realized that 2 cents wasn’t really worth it beyond the initial thrill of it all.

Like Daniel, I then encountered what later became obvious as a scam: someone who said they’d pay me money for signing up, but the time given was too fast and there was no way to actually verify that I completed the task in the way they said I needed to. This made me angry. How in the world does Amazon – one of my favorite companies on the planet, by the way – let them get away with this? I reported the bastards, but I don’t know what happened to that complaint. A few days later, Mechanical Turk still seems littered with these scams. That’s disappointing.

I turned to a final HIT, what seemed like just an individual who wanted me to go to his website and post any kind of comment, just for the sake of getting more hits. Seemed a little shady, but I gave it a try and duly posted a comment. Days later, I got the bad news: rejected! I wouldn’t be getting the shiny nickel I was promised. Hey Amazon: how is that allowed?

I was once so excited by the great name and the prospect of making easy money, but I'm now deeply disappointed with both the requesters and with Amazon. Why did all the posters seem so scammy? Why aren't there any tasks that seem interesting and worthwhile? And how does Amazon let the requesters get away with this? I felt somehow betrayed by an Internet company that I feel oddly loyal to; it was as if HBO launched a new channel that turned out to be entirely infomercials. Get your act together, Mr. Bezos!

I logged into my account a few days later. The final tally was 8 HITs accepted; 4 submitted; 1 returned; 3 abandoned; 3 approved; 1 rejected; and, in the end, 6 measly American cents earned. I wish that the original chess-playing Mechanical Turk hadn’t been destroyed in 1854. That Mechanical Turk might have been a sight to see. This one isn’t.

Lien Ceulemans

After all the Christmas presents I had to buy for my family, I thought that a good way to earn some money (while doing my homework) was to volunteer as a mechanical turk. A Google search brought me to the Amazon mechanical turk website (https://www.mturk.com/mturk/welcome).

Before starting, I first had to create a payment account and accept the 6 pages long “Amazon Mechanical Turk Participation Agreement”, thereby exclusively granting my intellectual property rights to the “Requester”. Furthermore, this agreement informed me that, as a mechanical turk, I would perform a “work for hire” “and would “not be entitled […] to any vacation pay, sick leave, insurance programs, including group health insurance or retirement benefits; or […] worker's compensation benefits in the event of injury.”

Amazon describes its platform as “a marketplace for work that requires human intelligence” There is indeed no sign of a worldwide crisis, as 306,354 “Human Intelligence Tasks” or “HIT’s” were available.

I started with an easy, quite boring assignment that would pay me 0.05 dollar for each correct hit: “Find the Website Address for a Restaurant and its Menu”. Since at that time, my “value” was still rated at 100, I met the qualification requirement. While the website of “Merlo La Trattoria” in Chicago is quite easy to find, I ran into difficulties after a few hits. The address of the “Brazil steak” house was only indicated in Chinese characters in the assignment, so I got completely lost at Google. It was time for another challenge…

For my next HIT, I had to “write a short answer about cars” of 50 to 60 words. Since I do not know anything about cars, this seemed the perfect HIT to test the reliability of the mechanical turk platform. To my best knowledge, I answered the question “How do I clean a mass air flow sensor on my Mercury?” as follows: “This is very easy to clean. You just take some water and soap that smells delicious and pour the water into the mass air flow sensor. If it is still not clean enough, repeat this process for a couple of times. You will see that your mass air flow sensor will look brand new again.” Although the HIT explanation mentioned that no qualification was required, after performing the hit, my answer was “submitted to conjecture corporation and [would] be approved or rejected shortly.” A few hours later, I received a “Qualification Request Rejection Notification” email ‏for Amazon Mechanical Turk. Despite I failed the test question, I did receive the 0.03 dollar reward for participating in this HIT.

After this, I decided to find a HIT which was more related to my field of knowledge. For different law schools, I needed to find the final fall application deadline for 2010 and the URL of website where I found the information. There were still 53 HITS available with a reward of 0.05 dollar each. Although the HIT information indicated that each HIT would take me around 10 minutes, I managed to complete a single HIT in about 4 minutes. This would thus result in an hourly wage of 0.75 dollar. Taking into account both the operating costs (e.g. electricity, internet, etc.) and the opportunity costs involved, I wonder who - besides “Difficult Problems in Cyberlaw” students - is willing to be a mechanical turk.

Although I got very bored, I tried to perform another HIT that required me to find the date, parties and subject of so-called “top secret letters of the US war department”. While this sounds as a very exciting and interesting assignment, in reality, trying to analyze unreadable pdf’s was not that fun. I decided to give it a last try, but had a hard time to find any attractive assignment. In the end, I participated in a research of the UNC Charlotte Human Computer Interaction Lab that does research in policy making decisions for social applications. After giving my ‘Informed Consent for Social Application Decision Making”, I had to indicate whether (i) I would or would not accept a certain Facebook application and (ii) share information of my Facebook friends with this application. Although this was not really interesting, this HIT gave me at least a good feeling as I contributed to university research. This was perfect to end my life as a mechanical turk.

Today (i.e. 5 days after submitted more than 20 hits), my Amazon Payment Account indicates that the approval of my submitted HIT’s is still pending. This means that so far, I only earned 0.03 dollars for 2 hours of labor.

Juan Cheng

I signed up to Mechanical Turk to complete a HIT. Before I decided which one to work on, I browsed the available HITs to see what kind of jobs are offered there. Because I wanted to know what is the difference between the HITs and the tasks in physical work environment. I found most of the HITs are trivial, boring and kind of meaningless. One of the HITs even highlighted that “this job is only for [a person's name]. Anyone else will be rejected”. This is so weird. Why the requester just ask the person he designate to finish the job offline?

I finally decided to work on a HIT which asked the worker to find the official website address for a restaurant and its menu. The requester provided the full name, address and telephone number of the restaurant to help target the object. It's pretty easy. I just surfed on Google and found the website address in 2 minutes. As requested, I attached the URL of the website address and that of the menu and submitted the HITs on December 28, 2009. Then I returned to my account to see the status. It showed that the HIT I submitted was pending for approval/rejection by the requester. Up to now, I have not been approved or rejected yet. I've sent a email to the requester to check. Will follow up.

One thing to note is before I selected the HIT above, I accepted several other HITs to try to work on them. I then returned them because they were very boring and time consuming. I supposed Mechanical Turk would not made a record for this. However, they actually did and even recorded the submitted, returned and abandoned rates based on the total HITs I accepted. It makes me feel like I'm an employee of Mechanical Turk and they will probably evaluate my performance based on these rates and other data, e.g. the feedback from the requester.

I guess the requester offered this HIT for at least two purposes. One is to get more people know the restaurant. Actually I really like the style of the website and I would try this restaurant if I happen to be around. The other might be to get more people search it on Google. If there are plenty of search on this term, the name of the restaurant appears as a google suggest.

Victoria Baranetsky

So fascinating. I “played” on Mechanical Turk for several days doing a variety of hits. I had never participated on Mechanical Turk before but soon became a junkie. Spending an embarrassing amount of time on it. Some tasks kept me entertained longer then others. The ones I like I would repeat over and over. And when I got bored with one I moved onto the next. Here is a short list of a few of the HITs that I participated in.

The first was a psychological study, I assume. The task was to rate the attractiveness of people in 30 photos from 1 (least attractive to 10 (most attractive).

The second was called “Basketball Tracking.” As footage of a basketball game played I was assigned to move a box over the image of the player. (A somewhat simpler version of Duck Hunt).

The third was named “Tag Fashion Magazine Covers.” As you clicked through different covers of beauty magazines the task was to categorize the predominant color on the cover (e.g. blue, green or red), the “look” (e.g. sexy, glamorous, or charming) and the brand of magazine (e.g. Vogue, Cosmo or Hustler).

The fourth was a photo identifier of cars. A car listing including the model and year was provided and the HIT required that the Turker find a matching photo for that specific car.

After I did these HITs I wanted to do something a little more thought provoking so I took three qualifying exams that were simple multiple choice questions including my place of birth, age and degree of education.

In total there are several things that struck me while “playing” with mechanical Turk.

First, the diversity of HITs was fascinating. The HITS were all over the board – going to show the incredible ability for Turk to be a great tool for a plethora of organizations.

However, although the diversity was phenomenal I was a bit hesitant with every HIT because I didn’t know the purpose or reason for what I was doing. For example, the psychological study that asked me to rate the beauty of the persons in the photographs was bizarre in asking me for a waiver. The waiver explained that I was not allowed to bring suit for my work in this study and that I had no monetary rights.

Not only was the purpose concealed but the member asking for the HIT is not revealed. Although there is a name for example “Paul Goldstein” is listed as the requester – further information is not available. I think this lack of transparency is troublesome. Anonymity certainly promotes more use of Mechanical Turk but it seems when some tasks could perhaps have malicious purposes, transparency would be good. In addition, transparency of the Turkers also would make their work more veritable.

Lastly, the final thing that troubled me on Mechanical Turk was the addictive quality of the tasks. By placing money into the equation one feels that playing, categorizing and writing reviews - in essence what is wasting time is perhaps a valuable to spend time. The games become frighteningly more appealing and what begins as a 45-minute endeavor turns into several hours of clicking, tagging and reading. I felt like a better at Atlantic City – pulling the proverbial lever at the slot machine over and over again.

Andrew Jacobs

I also experimented with Amazon’s Mechanical Turk.

My first HIT was to transcribe a 30 second audio clip. From the perspective of a new mturk worker, I was entirely satisfied with the experience. I could preview the audio clip before accepting the HIT, so as not to commit myself to garbled mumbling and subsequently damage my reputation by abandoning the task. The content turned out to be a snippet of Biblical interpretation. I accepted, did my best (bracketing a few words I wasn’t sure of, including “Millenialists”), and a few days later received two cents in my payments account for approximately three minutes of work.

From a slightly more critical perspective, this HIT got me thinking about economic and noneconomic incentives. As an unpaid intern for a literary magazine a few years ago, I transcribed a two-hour roundtable discussion among five barely distinguishable voices, recorded on a cheap, analog tape recorder. Though the task was awful, I worked on it diligently, mostly because I liked the editors and believed in their cause. If that alleged “Millenialist” word had popped up, I certainly would have gone to greater lengths to verify it than I did here, even though I risked two whole cents in this case. What if, then, there were a volunteer crowdsourcing tool for this requester’s particular brand of Christianity? Or, more plausibly, a general volunteer crowdsourcing website that grouped unpaid tasks by the causes that their completion furthered? The results might well be of higher quality than mturk’s, and I imagine those volunteer sites would avoid some thorny labor issues that Amazon faces.

My second HIT, which a few others have mentioned, was easy: I was to find the URLs for the main website and the menu of a given restaurant, at five cents a pop. (If either of these websites proved nonexistent, there was space to say so, and I still got my nickel.) Though it took a couple more days than the first HIT, I was eventually paid $.15 for my three finds.

The mindless, but not grating, simplicity of this HIT made me think a bit about how mturk’s economic incentives may be better utilized without increasing their value. In order to shift my earning to an Amazon Gift Certificate, for example, I have to follow a handful of steps and type in exactly how much I want to transfer. Why can’t I have my account be directly and continuously tied to a Gift Certificate? And why not have a “HIT it” button along with “Buy” in Amazon’s retail stores? If there wasn’t enough money in my Payments account, the button would take me to my mturk account so I could earn some more. One of the most valuable things I can get for less than a dollar, in my mind, is an Amazon MP3. If I’m working for tiny sums anyway, I might be more motivated if my tiny carrot were a more direct embodiment of value.

Now that I have a Mechanical Turk account, will I use it again? Maybe. Like Victoria, I had some fun completing the “Tag Fashion Covers” HITs (though at first the “submit” button in their dialog box was malfunctioning). Perhaps even as much fun as, say, those last twenty minutes of bopping around on the Internet before bed. I smell a New Year’s resolution. . .

Elisabeth Oppenheimer

I too spent some time on Mechanical Turk. I'd used it a few times before, which was enough to teach me to stay far away from $.02 tasks. They're boring, they often just don't work, and they pay $.02. But I actually enjoy some of the higher-paid tasks, especially writing articles. So I sorted the available HITs by wage and started with the one that paid the most, $4.17/hour. That seemed like an unbelievably high wage, particularly since it only asked me to look at a website and review it. After I'd glanced at the website, though, I realized that it was impossible to submit the HIT. Clever spam: the highest-priced task gets lots of viewers, who click through to the ad-loaded website, whose operators are presumably paid per view (or just hope some Turkers will click through). The operators didn’t any up paying for any HITs, so voila—free traffic.

Moderately disgruntled, I found a $1.70 HIT to write a 300 word piece about green weddings. It took maybe 5 minutes to complete, and I had fun--law students don't get to do much free writing, and since my friends are getting married in droves, I have a lot of thoughts about weddings. The payment came through the next day. I also wrote an article about how to make airline travel quicker and easier ($1.00, 300 words), also quickly paid. Then I started on a HIT that asked me to write a 500 word article including the words "sheer bodystocking." I wrote the HIT about something completely unrelated to lingerie (I wrote about sledding in my hometown), added the words "sheer bodystocking" in the middle of a sentence, and submitted. I wish I could say this had been a conscious quality-control experiment, but I just wasn’t paying much attention when I started writing. My article been pending several days, and I’m eager to see whether I get my $1.00.

Like most of you, I assume that these articles are not in fact serving those who want to know how to plan a green wedding or facilitate air travel, but are ending up as word salad on spammy webpages. (Despite that knowledge, I find myself surprisingly invested in writing good and useful articles – irrational?) So I saved my text and did a Google search for it a few days later, and got nothing; I’ll repeat it in a few weeks. As much as I find AMT tasks strangely soothing, like many of you, I don't think they always make the internet a better place.

I also paid a brief visit to the Turker forums, where there's a discussion going on about the perceived increase in spammy HITs, whether reporting does any good, whether companies will find out who reported them and reject that person's HITs, whether Amazon should preemptively screen HITs, and whether class-action lawsuits might do any good. Interestingly, a few requestors were participating in the discussion too, and one experienced Turker appeared to be in conversations with Amazon's management.

Finally (as discussed a little more below), I set up a TwitHawk alert for people tweeting about Amazon Mechanical Turk. I haven’t gotten any Turkers yet, but I did get requestors advertising HITs and a couple of articles about the service. One article made a good point--Amazon has long relied on customer rankings and data to create a huge part of the service's value (imagine how lame Amazon would be if only staff could do reviews!), so it was natural they would think to expand into the crowdsourcing space.

LiveOps

Sheel Tyle

I am in the process of signing up as an inexperienced bilingual (Spanish & English - I'm conversational, not fluent, in Spanish, but let's see how much I'm tested on it) LiveOps call center agent. There are videos on the website from independent agents, testimonials on the sidebar ($15-22/hour), and press releases from various periodicals that try and convince.

There are five steps that I must follow in order to be 'submitted for review':

Validate your email address

Verify your understanding of the general requirements

Provide basic information on your background

Assess your comprehension and computer skill

Audition your voice

Under 'basic information', here are some of the questions:

Have you ever contracted your business with LiveOps in the past?, Do you have prior call center experience?, Are you currently licensed to sell both health and life insurance products?, Are you currently licensed to sell both property and casualty insurance products?, Do you have prior experience in sales?, Do you have prior experience taking calls from Radio offers?, Do you have any experience in outbound telemarketing?, Do You Speak, Read and Write Spanish - FLUENTLY?, Do You Speak, Read and Write French - FLUENTLY?

I said "No" to every one except "Yes" to speaking Spanish.

Then, under 'comprehension and computer skill', I had to answer questions like:

Please read this script:"We dont want you to miss out on this great offer, so what I can do for you today is offer you 1 HotBrush for 3 easy payments of $29.99 plus $14.75 to cover processing or you can take advantage of our special offer, the SpeedyHeat model for only $6.67 additional per payment. The SpeedyHeat Model contains a computer chip which lets your HotBrush heat up faster and hold a more even temperature just like the most expensive professional quality hot tools. So, would you like to order the HotBrush or the SpeedyHeat Model?"

In this script, is the customer choosing between two different products or are they deciding whether to add a product (for two products total)? Deciding whether to add a product Choosing between two different products

What is the keyboard shortcut to move between windows you already have open on your computer? Hold down the "Alt" button and press the "Tab" button Drag the window offscreen Click the appropriate button in the task bar Close the program you are currently working in, and open a new one Minimize the window you are currently working in

Finally, the voice test. LiveOps asked me to call in to a 1-800 number and, when prompted, read two passages: one in English and one in Spanish.

Submitted. Once I hear back, I'll update whether I was accepted =)

Ramesh Nagarajan

I've completed the agent qualification process to become a LiveOps representative. Sheel did a good job of explaining the basics of the qualification process, so I'll share two observations I had about the process.

First, there's a fair amount of legal language LiveOps uses, with the intention of disclaiming any liability or even the existence of much of a relationship between it and its contractors. One must agree to the following: "I understand that LiveOps will investigate all of my information provided during the Agent Qualification process, and that Certification of my home business to contract with LiveOps will be contingent upon successful completion of a criminal and credit background check." They aren't currently accepting Massachusetts residents as LiveOps contractors -- perhaps Massachusetts's laws are too worker-friendly -- so I hope that means I get to avoid the "criminal and credit background check." Also, I didn't have to provide a Social Security Number, which makes me wonder if there is a real background check. Going back to the phrase I agreed to, I found it interesting that LiveOps representatives are setting up "home business[es] to contract" with LiveOps. There seem to be at least two advantages that LiveOps has over traditional telemarketing and call center companies -- first, it could save money on infrastructure by having employees work out of their own homes and use their own computers and phones, and second, it could save money by not having actual employees. I wonder which is more important, and if the second is a necessary part of the company's business strategy.

On a lighter note, the questions for "comprehension and computer skill" were quite entertaining. I was asked if booting a computer meant to turn on the sound, throw it out, turn it on, or add extra drives, and I had to decipher the meaning of a call script. I think there's a good chance I got one of the questions wrong. It asked what a customer paid today for a product that had "an upfront trial payment" of $15, three monthly payments of $40, and a shipping and handling charge of $10. I answered $15, but when I think about it, maybe it should be $25.

Online mass collaboration project

Franny Lee

While waiting for my little sister to finish teaching her last Grade 9 class of 2009, I fell into conversation at an Ottawa (Canada) Starbucks with Russell Maier, a professional collaborative multi-media artist and a fellow fan of peppermint mochas. Russell is currently cycling around the world to help promote and orchestrate a “planetary collaborative mosaic” art/photo project, based upon the ancient cultural concept of a “mandala”. The first grand “1Mandala” is slated to be unveiled in front of the NYC UN Headquarters in May 2010.

The project’s theme - global peace and togetherness: “As a celebration of our Oneness we are building a collaborative planetary mandala. Our mosaic 1Mandala is being built out of pictures of people of all places and cultures smiling peace to the world. These "peace portraits" are being arranged into a Mandala-- an ancient circular and geometric art-form long-used by the world's spiritual traditions for healing, unity and raising consciousness… It will be a powerful symbol of humanity united in the intention of peace to each other and the planet. 

We invite you to share your peace and your portrait with the world.”

Photo contributions are accepted through the project's website at http://www.1mandala.org. From my conversation with Russell, I get the sense that artistic vision currently dominates the project. Issues of communicating his project to a global audience are forefront (in addition to the website, Russell’s blog of his travels http://missive452.blogspot.com/ and the “daily e-blasts” with the “peace portrait of the day”, this project participates in Facebook, Flickr, Twitter, Digg and StumbleUpon), while intellectual property or privacy concerns have not been addressed other than with a simple website privacy policy and release (see 2-paragraph “Privacy and Release” at http://www.1mandala.org/en/uploader). Photos are (or will be) screened by a member of his team at some point before the final UN premiere, however, Russell prefers not to interfere with individual contributions unless the photo is deemed absolutely inappropriate. His logistical vision is to ultimately develop a fully automated online photo contribution process. There have been very few (I believe less than 5) instances where a photo has been refused - if I recall correctly, one may have involved something about a turkey.

After creating my peace picture, I began the online contribution process and was required to register my email address, first name, city and country. After completing registration, I received an email with instructions for my photo contribution, which included general tips for “peace portraits” and a hyperlink to the contribution uploader webpage. From there, I completed the web-fillable form by uploading my peace portrait, advising of my city and country, and was given the option of adding a photo caption and including a story about my photo. A box for the privacy policy and release and a box to confirm that this is an “intentional peace portrait” must be checked off to proceed to the next steps, which are to crop the photo to the project size as per the instructions provided, and finally, to submit the photo. I kept experiencing “Internal Server Error” messages while trying to submit my peace portrait, and finally contributed through the alternate method of emailing a copy to peace@1mandala.org.

Amanda Peyton

Nov. 4, 2008 is a day that brought both metaphorical and tangible change to the White House. For many Americans, the day holds special, "memorable" significance.

A young filmmaker at IFC named Jeff Deutchman decided that the day would be worth capturing (regardless of the outcome of the election) in many different cities, so he enlisted a few friends to film highlights from Election Day that he then spent a few months editing into a proper documentary.

But then he realized that by only using the footage from his friends, he was ignoring what was surely massive amounts of footage and still photos from those outside of his personal circle. He started a website, www.11-4-08.com, where people can browse, upload and edit footage from November 4th into their own "documentaries". The participatory site will launch in conjunction with the film's screening at a prestigious film festival in March.

The website accomplishes several objectives, and touches upon a few themes of mass collaboration:

1. If people watch the documentary and enjoy it, they can continue to interact with the film through the website

2. It allows history to become personal - while there were certain events on Nov. 4 that everyone in the country shared, this website - like many popular tools on the web, allows the user to personalize their experience.

3. It connects people who previously would have had no interaction by letting them access, watch and edit each other's footage

My involvement with this project has been to brainstorm how to structure the crowdsourced part to ensure ease of use and maximum participation. When putting together a site where the first objective is to prompt some sort of participation from the user (as opposed to straight consumption) there are several elements that I believe are critical:

- Clear objective of what you want from the user. For this project it's clear - to contribute still photos and footage from Nov. 4, 2008. I believe that this request is focused enough that the user will not be overwhelmed. If you make your requests too broad often times the user will get overwhelmed.

- Recognition. A large motivator for those who participate in crowdsourced projects is the incentive structure. How will the participants be recognized? (This could be a separate essay in itself - Amazon, Yelp, Reddit, Stack Overflow and Hacker News all boast their own incentive structures to encourage participation).

- Speed and Tech - the back-end technology needs to be seamless. For this project, the challenge has been to identify an in-browser simple video editor that will allow people to take existing footage and edit it into their own mini-documentaries. Currently, video editing software is large and bulky, but there are a few companies that are building simple browser-based video editing tools. Also, as Google has taught us, the site needs to be fast. The fastest way to scare away users is to have a slow site (just ask Friendster).

When the film is shown in March and the project launches, I am very curious to see how people will use the site and how participation will materialize.

Sharona Hakimi

A November New York Times article http://www.nytimes.com/2009/11/17/technology/internet/17maps.html?_r=1&nl=technology&emc=techupdateema1 about a wikipedia-style mass collaborative mapping effort sparked my interest in various grassroots and industry induced efforts to find the best and most helpful way to "map"the world around us. We all use google maps, and with the satellite images and street-view capabilities, google maps has helped define the way I travel now, especially as they become readily available on our cellphones. However, as the article discusses, mapmaking is largely political, and left solely to google experts to decide what to map, how quickly, and from which views, we are leaving it to strangers with a vested interest. Hence, organizations like OpenStreetMap and wikimapia have come about in an effort to democratize mapmaking, leaving it in the hands of reliable locals to show just where the nearest coffee shop or library is.

I played around with three different mapmaking websites (one of which is sponsored by google, and another which partners with it to lay your data on top of google maps).

The first website I tried is OpenStreetMap (http://www.openstreetmap.org/) which is a non profit aimed at making free maps that can be reused by anyone. According to NYTimes, it has some 180,000 contributors who have mapped many countries in varying levels of detail, and it even has its own twitter account. Although I love the idea behind the website, I think the layout is actually really confusing, and the search is terrible. It's set up as a wiki, allowing contributors to edit and discuss various changes. It was a bit of a hassle to sign up for an account, and any contributions have to be authorized, so I have yet to see if my additions to my neighborhood in Toronto (a pharmacy, a synagogue, etc), were added.

The next website I tried is wikimapia (http://wikimapia.org), which partners with google maps, allowing users to create specialized maps on top of google. I spent some time trying to label buildings at Harvard Law school, but when I actually tried confirming the changes it said it was already labeled (though it didn't seem to be). Overall I found this website a lot easier to use and I like the layout more, although the search function is also not very good.

The last one I used was Google Map Maker (http://www.google.com/mapmaker), which immediately opens a very user friendly help box explaining the user's role ("Your work as a citizen cartographer will appear on Google Maps and serve the needs of local users, schools, city planners, tourists, and more."). Although I hate to say it, since this "democratization" of mapping is initiated by a for-profit industry member, this website is by far the most advanced and user friendly, and I found it really fun.

Overall I'm a really big fan of the democratization of mapmaking, and I think it fits with the general concept to have various websites that function in different ways to do the same thing. As the technology becomes more advanced, I think all three websites will get even better. I definitely believe that mass collaboration is extremely important and even necessary in making making maps in the 21st century.

TwitHawk

Elisabeth Oppenheimer

(Formerly TwitterHawk, but Twitter made them change the name.)

The TwitHawk space looked forlorn, so I took a quick look at it. It's pretty clever. You set up a search -- say, people tweeting about Amazon Mechanical Turk within 25 miles of Palo Alto -- and the app updates you when someone tweets on that topic. You also set up some prepared responses ("I’m studying Amazon Mechanical Turk for a class, how has your experience with it been?") which are automatically sent to anyone who matches your search. You can also change the settings such that you need to confirm before sending your message, or so that you automatically follow anyone you message. It's basically a marketing tool: in the example they give, a coffee shop searches for local people tweeting about a desire for coffee, and then sends them the address of the shop.

The concern in the press coverage is that this will massively spam Twitter users, but the owners have taken some useful steps to prevent this--you have to pay 5 cents per message, you can only send one message to a given user, and you can only send one message every two hours.

I'm impressed with the service, although I’m still fundamentally stunned that Twitter and Twitter apps have morphed so quickly from a "huh? why would you want to do that?" phenomenon to a "cornerstone of all cool marketing campaigns" phenomenon.

Reuben Rodriguez

First things first. As mentioned above, Twitterhawk is dead. Long live Twithawk! I think Elisabeth did a pretty good job of describing what Twithawk does so I'll keep my description to the bare minimum: Twithawk is basically a tool that sends preset replies to other Twitter users whose tweets match whatever search results you have set up. Twithawk gives new registrants 10 free auto-replies, so I set up a new Twitter account and gave it a spin.

I decided I would be tweeting as HarryDoyle97 and accordingly constructed a search for any mention of the word "Indians" within 25 miles of the Cleveland area. I created three pre-set responses and set up the search to run once every two hours. Within 10 minutes, Twithawk had sent out my first auto-response (JUST a bit outside) to Indians on Fan Feedr in Cleveland, Ohio. It was all very easy.

Elisabeth discusses the critics' main concern about how Twithawk could be used for mass Twitter spam and she mentions the steps Twithawk has implemented to try and combat this possibility - the five cent per reply cost and the limited frequency with which a response can be sent. There is also subtle encouragement to use the Twitter feed as more than just a robot automatically replying to search results. Each "campaign" that you set up has a meter on your dashboard labeled "TwitterHawk Noise Ratio" with an arrow pointing from green to red. It then rates the ratio between Twithawk responses on your feed vs. "natural" (read: by an actual human being) responses and encourages the user to keep the ration below 25% (1 Twithawk response for every 4 real ones). While this may not deter those with the real intent to spam, at the least it serves as a helpful reminder to the well-meaning, but inattentive marketer to keep their feeds from being mere spam. Finally, the FAQs asks those who feel that Twithawk is being used for spam to report the spammers and notes that Twitter is active in punishing spammers. If Twithawk remains vigilant, the product definitely has potential.

Tyler Lacey

I used TwitHark too. My goal was to find people complaining or wondering about on the status of the (often late) Boston subway and bus system called "The T" and run by the MBTA. So my search was simple: I merely searched for terms such as "MBTA", "T", "late", "delayed", etc. that posted within 5 miles of Boston. My response was to tell people about the official MBTA twitter feed (MBTANow) that already supplies people with very up to date information. I set TwitHawk to send this response automatically.

I suspect that my response will annoy more people (at most 10 really, since I am only using my ten free responses) than it helps for two reasons. The first is that "within 5 miles of Boston" is too coarse of a search area. It will likely define lots of people that do not or can not use the T and exclude many people that do use the T. This difficulty is a symptom of the my second, more general, complaint with TwitHawk's search mechanism: it is too simplistic. For something as delicate as searching the millions of twitter posts and responsing in a useful manner, I think I need more than the simple list of terms that are presumably searched for in an "or" fashion, narrowed by a single location. I would prefer something more akin to regular expressions (http://en.wikipedia.org/wiki/Regular_expression) that would let me specify precisely the strings that I am searching for, as well as a more precise manner of specifying locations (so I could use something like "within 1 mile [Town]" for every town near Boston that the T serves). This way I would be able to target posts that are more likely complaning or inquiring about the status of the T and search within an area that more closely resembles the T network rather than a circle extending out from the center of Boston. I believe that this advanced search capacity should be built into TwitHawk using a more sophisticated search builder interface. This would allow users not familiar with the complexities of regular expressions to make use of their flexibility. However, an API or program that sat on top of TwitHawk could achieve a similar result by flattening a regular expression into many simpler searches and automatically entering those searches in TwitHawk. However, for complicated regular expression searches this would create an unmanageable number of simpler searches for TwitHawk to store.

I also noticed that a captcha is the only thing that seemed to be standing between me and multiplying my ten free opportunities to spam into millions of free spam messages by automatically creating TwitHawk accounts and setting them to the same searches. Without knowing the internals of TwitHawk I can't be sure what defenses exist to prevent this.

Drumbeat Privacy Project

Michael Feldman

Ok. So I was intimidated by some of the other possible projects such as Twithawk and Mechanical Turk. Signing up for things and following directions? Ew. So, the clear solution was to try and tackle the creative problem proposed by Drumbeat to simplify various internet companies' privacy policies, a la Creative Commons. The idea is to create a simple structure of icons, images and phrases to more effectively communicate a companies privacy policy without requiring the user to sift through a ton of dense legalese.

In creative commons, each icon which stands for a particular set of rights the author wishes to protect. But this won't work for privacy policies because the policies are different for each company. They don't merely invoke a set of already-delineated rights. One organization called Privacy Choice has already tried to create icons to simplify the process of learning about companies privacy policies, (see http://www.privacychoice.org/whos_watching). But this website is not completely effective since it requires the user to "mouse over" various categories of privacy policies, only to encounter much more "mumbo jumbo."

It seemed to me that the primary design problem was that there are multiple levels of information: specific data within specific provisions within general categories. So I tried to create an idea for a basic system which used the Privacy Choice Categories (plus a few others I thought would be relevant), color coded them so that a user could tell which categories might require greater attention, and then placed the provisions on a rather vague scale. While not as specific as an ultimate solution might want to be, this at least gives a user site-specific information that will be useful before delving into the actual text. From here, the next step would be to create a stock of short phrases which could appear when the user mouses over the icons which would give more detailed information without resorting to large blocks of text.

All in all, the challenge is very interesting and definitely something I would be happy to continue to work on over the next few weeks. The specific challenge of conveying all information in a given privacy policy in simpler symbols is one which will likely require involvement of the actual companies to conform their policies to set rubrics (or at least stock modular blocks of text), but short of that, it might be possible to create a more intuitive set of symbols that could be more generally applicable to all policies. But then there would still be the problem of aggregation of information since people would have to go through each policy to decode it and re-encode it into the new system.

You can find the actual suggestions I posted to drumbeat here: https://wiki.mozilla.org/Talk:Drumbeat/Challenges/Privacy_Icons.

Bruno Magrani

If Firefox is about making the web better through software, drumbeat intends to be the twin brother that focus on connecting project ideas and people together to promote a better web. In a way, one can think about it as an incubator for projects to improve people's experience on the web.

Because the project seemed quite exciting to me I took the liberty to exchange some emails with the people involved to get a sense of how I could be of most help to the project and decided to collaborate coming up with other examples of people who are contributing to improve the web. This first initial experience – being able to chose how to best participate in the project – denoted one of the marvelous of online peer production: the advantage over traditional industrial information production systems in terms of information gains regarding how to connect a person's interest with a specific task.

If one important component of the project involves coming up with ideas for projects an equally important component involves getting real life examples of who is doing what to improve the web. It works both as way to recognize the efforts of these people, but also to stimulate other people to do the same and start their own initiatives. In a sense this is intended to send a message saying: everyone can and should contribute to make a better web.

One of the things that I notice while reading the projects and people being featured was that they were too much focused on experiences happening in the United States and some in Europe, so I decided to come up with examples of what both common and notorious people were doing in a developing country such as Brazil. My contribution can be found here and I'm definitely going to keep contributing to the project.