Crowdsourcing - Revision history

BerkmanSysop: UTurn to 1297036800

2020-08-13T10:46:23Z

UTurn to 1297036800

119.82.250.34 at 16:24, 6 June 2020

2020-06-06T16:24:15Z

← Older revision		Revision as of 16:24, 6 June 2020
Line 148:		Line 148:
	''Solutions''		''Solutions''

	*1. Platform-specific solution. The Best Practices document suggests that each crowdsourcing platform implement its own measures to track reputation. This is a reasonable solution. 99designs and InnoCentive do not yet have such a system. This kind of solution would allow each platform to identify the reputational characteristics necessary for the kind of work offered. 99designs, for example, could track worker reputation based on cooperation among workers, number of contests won, and portfolio ratings from other users. It could also track employer reputation based on payment amount and speed, and copyright licensing terms. InnoCentive, by contrast, could focus on credentials characteristics, such as degrees obtained or relevant work experience. It could track employers based on potential for future projects with the same company. The Best Practices document also suggests that workers have access to ~~their~~ data, and presumably envisions a way to use reputation information at one provider or another. Here, the solution could be for each platform to issue a "portable reputation," which would explain the reputation of the worker on that site, the work performed, and the average/median rating of a worker on the site (with similar work performed). In this scenario, platforms would have to have some interoperability, a problem on which some already are working.[http://portal.acm.org/citation.cfm?id=1839707.1839723] Each platform, for example, could agree to host the reputational information provided by another. In such a situation, the worker could 'import' his or her data from, say, 99designs to iStockphoto. When that worker completes a task or uploads a photo, an employer could click on the worker's profile to view various bits of reputational information. (Perhaps they could all agree to a specific file (type) that included certain information?) In one iteration, this could include "Reputation Homepage" that displayed the logos of various platforms for which the worker had completed tasks. Scrolling the mouse over the logo might reveal some general reputational information,such as a reputation score (with the appropriate scale for the platform). This idea, though, requires cooperation among platforms--something that has been difficult in the past. One reason for this difficulty is because large, established platforms have a perverse incentive to keep a reputation system closed. These platforms already have a user base that depends upon the platform. Portability threatens what they perceive as theirs: the reputations earned through their platforms, which make the platform a desireable place to conduct business. eBay, for example, objected when Amazon allowed users to import their eBay reputations, claiming proprietary interest.[http://www.unpcdc.org/media/6406/reputation%20mechanisms%20and%20electronic%20markets%20-%20economic%20issues%20and%20proposals%20for%20public%20procurement.pdf] (p. 237.) This confirms Randal Picker's contention that, as far as portability on the Web goes, "law matters."[http://www.law.northwestern.edu/lawreview/colloquy/2008/25/LRColl2008n25Picker.pdf] (p. 8.) [http://sandeepkumar.org/my/papers/2009_CAT_PortableReputation.pdf][http://www.igi-global.com/Bookstore/Chapter.aspx?TitleId=30454]		*1. Platform-specific solution. The Best Practices document suggests that each crowdsourcing platform implement its own measures to track reputation. This is a reasonable solution. 99designs and InnoCentive do not yet have such a system. This kind of solution would allow each platform to identify the reputational characteristics necessary for the kind of work offered. 99designs, for example, could track worker reputation based on cooperation among workers, number of contests won, and portfolio ratings from other users. It could also track employer reputation based on payment amount and speed, and copyright licensing terms. InnoCentive, by contrast, could focus on credentials characteristics, such as degrees obtained or relevant work experience. It could track employers based on potential for future projects with the same company. The Best Practices document also suggests that workers have access to [https://sbobet.gold/ Sbobet] data, and presumably envisions a way to use reputation information at one provider or another. Here, the solution could be for each platform to issue a "portable reputation," which would explain the reputation of the worker on that site, the work performed, and the average/median rating of a worker on the site (with similar work performed). In this scenario, platforms would have to have some interoperability, a problem on which some already are working.[http://portal.acm.org/citation.cfm?id=1839707.1839723] Each platform, for example, could agree to host the reputational information provided by another. In such a situation, the worker could 'import' his or her data from, say, 99designs to iStockphoto. When that worker completes a task or uploads a photo, an employer could click on the worker's profile to view various bits of reputational information. (Perhaps they could all agree to a specific file (type) that included certain information?) In one iteration, this could include "Reputation Homepage" that displayed the logos of various platforms for which the worker had completed tasks. Scrolling the mouse over the logo might reveal some general reputational information,such as a reputation score (with the appropriate scale for the platform). This idea, though, requires cooperation among platforms--something that has been difficult in the past. One reason for this difficulty is because large, established platforms have a perverse incentive to keep a reputation system closed. These platforms already have a user base that depends upon the platform. Portability threatens what they perceive as theirs: the reputations earned through their platforms, which make the platform a desireable place to conduct business. eBay, for example, objected when Amazon allowed users to import their eBay reputations, claiming proprietary interest.[http://www.unpcdc.org/media/6406/reputation%20mechanisms%20and%20electronic%20markets%20-%20economic%20issues%20and%20proposals%20for%20public%20procurement.pdf] (p. 237.) This confirms Randal Picker's contention that, as far as portability on the Web goes, "law matters."[http://www.law.northwestern.edu/lawreview/colloquy/2008/25/LRColl2008n25Picker.pdf] (p. 8.) [http://sandeepkumar.org/my/papers/2009_CAT_PortableReputation.pdf][http://www.igi-global.com/Bookstore/Chapter.aspx?TitleId=30454]

	*2. Uniform Reputation. Another approach to reputation would use a centralized entity to manage worker/employer reputation across all platforms. One can imagine a universal reputation platform that draws information from various sites and aggregates it--maybe it has agreements with all the platforms to provide it reputation-related information. (There are already sites that purport to provide reputational rankings of websites--see, e.g., Webutation.[http://www.webutation.net/#]) In other words, this entity would maintain a database of all crowdsourced employers/workers reputations. Reputational information would be aggregated automatically from all participating crowdsourcing platforms, which would automatically submit data to this entity. Theoretically, employers and workers could consult this platform to determine worker reputations. To enable this "reputation finding," the reputation database could contain various "categories" of workers and employees, and allow an individual to sort based on various reputational scores, which would be determined based on a variety of metrics and the information provided to the centralized entity by the crowdsourcing platforms. So, for example, a company could find someone with a high "creative reputation" (e.g., logo design) or someone with a high "engineering" rating. It may also allow users to disaggregate reputational information across platforms to see how the users has performed/behaved in each. Additionally, a central reputation-authority would allow people to move "with" their reputations and stay "anonymous"--they won't disclose their real identity. As noted, this all would require platforms to (automatically) submit information to this reputation database/entity. Because platforms (in the hypothetical world) would automatically submit worker reputation info to the site, there are essentially two ways the site could work at the level of individual reputations. Both methods of operation work from the premise that individuals and employers are searchable by reputation. Search results might display an aggregate reputation or a simple ranking with no reputational score accompanying it. The first way resembles the solution described above: once a user is selected, a visitor is taken to a reputation homepage that displays a worker/employer reputation from different platforms, and provides information about each. This provides at least one benefit for platforms: they don't have to generate interoperability--they can delegate reputation aggregation to one entity, which could probably collect and display the information more effectively. Two authors have already sketched a design for a similar system.[http://sandeepkumar.org/my/papers/2009_CAT_PortableReputation.pdf] A second method would be to for the reputation site to create its own "general" reputation for each worker based on the information it receives from platforms. We can imagine a situation in which workers or employers can ask the reputation site to rank workers based on specific, general, or a combination of characteristics. This would allow employers to sort workers by reputational characteristics they deem important to a particular type of work. Once sorted, employers could provide an open-call project to the narrowed crowd of workers. This system also would require the centralized agency to weight different sites or reputational attributes. This is a problem of reliability, discussed below. One other advantage of a unified reputational system would be allowing some user control over reputation. We can imagine a place in a user profile where the user can fill in information, provide explanations, or otherwise comment on past work experience and the like. (There may also be room for a centralized-decentralized approach, where the user claims authority over her reputation and distributes it how she wants. Jon Udell outlined such a concept in his recent talk at the Berkman Center.[http://cyber.law.harvard.edu/events/luncheon/2010/12/udell])		*2. Uniform Reputation. Another approach to reputation would use a centralized entity to manage worker/employer reputation across all platforms. One can imagine a universal reputation platform that draws information from various sites and aggregates it--maybe it has agreements with all the platforms to provide it reputation-related information. (There are already sites that purport to provide reputational rankings of websites--see, e.g., Webutation.[http://www.webutation.net/#]) In other words, this entity would maintain a database of all crowdsourced employers/workers reputations. Reputational information would be aggregated automatically from all participating crowdsourcing platforms, which would automatically submit data to this entity. Theoretically, employers and workers could consult this platform to determine worker reputations. To enable this "reputation finding," the reputation database could contain various "categories" of workers and employees, and allow an individual to sort based on various reputational scores, which would be determined based on a variety of metrics and the information provided to the centralized entity by the crowdsourcing platforms. So, for example, a company could find someone with a high "creative reputation" (e.g., logo design) or someone with a high "engineering" rating. It may also allow users to disaggregate reputational information across platforms to see how the users has performed/behaved in each. Additionally, a central reputation-authority would allow people to move "with" their reputations and stay "anonymous"--they won't disclose their real identity. As noted, this all would require platforms to (automatically) submit information to this reputation database/entity. Because platforms (in the hypothetical world) would automatically submit worker reputation info to the site, there are essentially two ways the site could work at the level of individual reputations. Both methods of operation work from the premise that individuals and employers are searchable by reputation. Search results might display an aggregate reputation or a simple ranking with no reputational score accompanying it. The first way resembles the solution described above: once a user is selected, a visitor is taken to a reputation homepage that displays a worker/employer reputation from different platforms, and provides information about each. This provides at least one benefit for platforms: they don't have to generate interoperability--they can delegate reputation aggregation to one entity, which could probably collect and display the information more effectively. Two authors have already sketched a design for a similar system.[http://sandeepkumar.org/my/papers/2009_CAT_PortableReputation.pdf] A second method would be to for the reputation site to create its own "general" reputation for each worker based on the information it receives from platforms. We can imagine a situation in which workers or employers can ask the reputation site to rank workers based on specific, general, or a combination of characteristics. This would allow employers to sort workers by reputational characteristics they deem important to a particular type of work. Once sorted, employers could provide an open-call project to the narrowed crowd of workers. This system also would require the centralized agency to weight different sites or reputational attributes. This is a problem of reliability, discussed below. One other advantage of a unified reputational system would be allowing some user control over reputation. We can imagine a place in a user profile where the user can fill in information, provide explanations, or otherwise comment on past work experience and the like. (There may also be room for a centralized-decentralized approach, where the user claims authority over her reputation and distributes it how she wants. Jon Udell outlined such a concept in his recent talk at the Berkman Center.[http://cyber.law.harvard.edu/events/luncheon/2010/12/udell])

119.82.250.34 at 15:41, 6 June 2020

2020-06-06T15:41:54Z

198.98.58.135 at 17:35, 1 March 2019

2019-03-01T17:35:06Z

← Older revision		Revision as of 17:35, 1 March 2019
Line 7:		Line 7:
	Although the idea of crowdsourcing--if not the word itself--has been around for many years, the Internet has made it much easier, cheaper, and efficient to harness the power of crowds. The power of crowds was popularized in 2004 when James Surowiecki published a book entitled, ''The Wisdom of Crowds.''[http://books.google.com/booksid=hHUsHOHqVzEC&printsec=frontcover&dq=the+wisdom+of+crowds&hl=en&src=bmrr&ei=T0DtTI2FGYP88AbA78GaAw&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCsQ6AEwAA#v=onepage&q&f=false] This book purported to show how large groups of people can, in many cases, be more effective at solving problems than experts. According to Surowiecki (2004: xiii), "under the right circumstances, groups are remarkably intelligent, and are often smarter than the smartest people in them." Two years later, journalist Jeff Howe coined the phrase "crowdsourcing" to refer to work that was performed by the "masses" online.[http://www.wired.com/wired/archive/14.06/crowds.html] Since Howe's article was published in 2006, numerous authors have written books on crowdsourcing, each choosing to focus on different aspects of the topic. Howe himself took up the topic in 2008, proclaiming crowdsourcing to be a panacea--a place where a perfect meritocracy could thrive.[http://books.google.com/books?id=ge_0LBOcwWsC&printsec=frontcover&dq=crowdsourcing&hl=en&ei=QinGTKS9AcGAlAeHtLX-AQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CC8Q6AEwAA#v=onepage&q&f=false] Howe examined crowdsourcing from a variety of perspectives: what benefits it can provide, what kinds of tasks it can accomplish, and the potential changes it may bring about. Howe's prognosis for crowdsourcing was positive--in it he saw many potential solutions and few potential problems. Others have followed Howe's lead in describing the benefits of crowdsourced work. Clay Shirky has published two books--''Here Comes Everybody'' (2008)[http://books.google.com/books?id=UNxU-2s2sQYC&printsec=frontcover&dq=clay+shirky+here+comes+everybody&hl=en&src=bmrr&ei=TEHtTMD6FIT58Aa54JVs&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCQQ6wEwAA#v=onepage&q&f=false] and ''Cognitive Surplus'' (2010)[http://books.google.com/books?id=_U1nQgAACAAJ&dq=cognitive+surplus&hl=en&src=bmrr&ei=f0XtTMnsFcOC8gaD4MGRAw&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCsQ6wEwAA]--in which he describes how technology does more than enable new tools, it also enables consumers to become collaborators and producers. Although Shirky's books are not expressly about crowdsourcing per se, they mirror the optimism Howe expresses, both in terms of collaborative enterprises and the Internet's power to enable them.		Although the idea of crowdsourcing--if not the word itself--has been around for many years, the Internet has made it much easier, cheaper, and efficient to harness the power of crowds. The power of crowds was popularized in 2004 when James Surowiecki published a book entitled, ''The Wisdom of Crowds.''[http://books.google.com/booksid=hHUsHOHqVzEC&printsec=frontcover&dq=the+wisdom+of+crowds&hl=en&src=bmrr&ei=T0DtTI2FGYP88AbA78GaAw&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCsQ6AEwAA#v=onepage&q&f=false] This book purported to show how large groups of people can, in many cases, be more effective at solving problems than experts. According to Surowiecki (2004: xiii), "under the right circumstances, groups are remarkably intelligent, and are often smarter than the smartest people in them." Two years later, journalist Jeff Howe coined the phrase "crowdsourcing" to refer to work that was performed by the "masses" online.[http://www.wired.com/wired/archive/14.06/crowds.html] Since Howe's article was published in 2006, numerous authors have written books on crowdsourcing, each choosing to focus on different aspects of the topic. Howe himself took up the topic in 2008, proclaiming crowdsourcing to be a panacea--a place where a perfect meritocracy could thrive.[http://books.google.com/books?id=ge_0LBOcwWsC&printsec=frontcover&dq=crowdsourcing&hl=en&ei=QinGTKS9AcGAlAeHtLX-AQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CC8Q6AEwAA#v=onepage&q&f=false] Howe examined crowdsourcing from a variety of perspectives: what benefits it can provide, what kinds of tasks it can accomplish, and the potential changes it may bring about. Howe's prognosis for crowdsourcing was positive--in it he saw many potential solutions and few potential problems. Others have followed Howe's lead in describing the benefits of crowdsourced work. Clay Shirky has published two books--''Here Comes Everybody'' (2008)[http://books.google.com/books?id=UNxU-2s2sQYC&printsec=frontcover&dq=clay+shirky+here+comes+everybody&hl=en&src=bmrr&ei=TEHtTMD6FIT58Aa54JVs&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCQQ6wEwAA#v=onepage&q&f=false] and ''Cognitive Surplus'' (2010)[http://books.google.com/books?id=_U1nQgAACAAJ&dq=cognitive+surplus&hl=en&src=bmrr&ei=f0XtTMnsFcOC8gaD4MGRAw&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCsQ6wEwAA]--in which he describes how technology does more than enable new tools, it also enables consumers to become collaborators and producers. Although Shirky's books are not expressly about crowdsourcing per se, they mirror the optimism Howe expresses, both in terms of collaborative enterprises and the Internet's power to enable them.

	These books have provoked an academic interest in finding out ''who'' is the crowd, or why the crowd moves the way it does. Some have looked at scientific crowdsourcing, asking what characteristics make someone a successful crowdworker/problem-solver.[http://www.hbs.edu/research/pdf/07-050.pdf] Part of answering that question, it turns out, requires asking why people attempt to be part of the innovating crowd in the first place. The authors of this study found that the crowd was highly educated. They also found heterogeneity in scientific interests, as well as monetary and intrinsic motivations, to be important drivers of "good" problem-solvers. Others have examined non-scientific endeavors and asked similar questions. One report, for example, examined workers on iStockphoto.[http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2159/1969] This report found that most iStock crowdworkers developing photographs were highly educated and motivated primarily by money. Jeff Howe, however, takes a different perspective as to crowdworkers' motivation: "There are . . . two shared attributes among almost all crowdsourcing projects: participants are not primarily motivated by money, and they’re donating their leisure hours to the cause. That is, they’re contributing their excess capacity, or 'spare cycles,' to indulge in something they love to do." (''Crowdsourcing'', pgs. 28-29.)		These books have provoked an academic interest in finding out ''who'' is the crowd, or why the crowd moves the way it does. Some have looked at scientific crowdsourcing, asking what characteristics make someone a successful crowdworker/problem-solver.[http://www.hbs.edu/research/pdf/07-050.pdf] Part of answering that question, it turns out, requires asking why people attempt to be part of the innovating crowd in the first place. The authors of this study found that the crowd was highly educated. They also found heterogeneity in scientific interests, as well as monetary and intrinsic motivations, to be important drivers of "good" problem-solvers [https://academicsciences.co.uk/about-us/]. Others have examined non-scientific endeavors and asked similar questions. One report, for example, examined workers on iStockphoto.[http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2159/1969] This report found that most iStock crowdworkers developing photographs were highly educated and motivated primarily by money. Jeff Howe, however, takes a different perspective as to crowdworkers' motivation: "There are . . . two shared attributes among almost all crowdsourcing projects: participants are not primarily motivated by money, and they’re donating their leisure hours to the cause. That is, they’re contributing their excess capacity, or 'spare cycles,' to indulge in something they love to do." (''Crowdsourcing'', pgs. 28-29.)

	While some focused on the potential consumer revolution or the composition of the crowd, others examined the business-related aspects of crowdsourcing. Identifying attributes of successful crowd innovators also has a business dimension. One researcher suggests that having experience spanning across a variety of communities or disciplines makes one likely to be considered 'innovative.'[http://74.125.155.132/scholar?q=cache:Ro0Ij3Y41wMJ:scholar.google.com/&hl=en&as_sdt=40000000&sciodt=40000000] Others focus more broadly on how to use the crowd to maintain or bolster business or brand. In ''Groundswell'' (2008),[http://books.google.com/books?id=YOVuQFXNcP4C&printsec=frontcover&dq=groundswell&hl=en&ei=ZF_tTKjtLIL58Ab-tu26AQ&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCsQ6wEwAA#v=onepage&q&f=false] Charlene Li and Josh Bernoff focus on how to most effectively use crowdsourcing to advantage businesses. The authors highlight how user bases of products can undermine a product or brand.[http://books.google.com/books?id=YOVuQFXNcP4C&printsec=frontcover&dq=groundswell&hl=en&src=bmrr&ei=tUPtTPOeAYP-8AbcxemdAQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCgQ6AEwAA#v=onepage&q&f=false] As a result, the authors propose that businesses use the "groundswell" to their advantage, fostering communities that can provide valuable feedback and economic payoffs. Marion K. Poetz and Martin Schreier have also taken a business perspective on crowdsourcing,[http://ssrn.com/abstract=1566903] arguing that the crowd is capable of producing valuable (but not always viable) business ideas at a low cost. Other researchers have found that young entrepreneurs who were attempting to start businesses frequently belonged to these kinds of communities.[http://ssrn.com/abstract=1022358] For a related discussion on user innovation and user communities, see Eric Von Hippel's books[http://web.mit.edu/evhippel/www/books.htm] and William Fisher's article in the Minnesota Law Review.[http://www.minnesotalawreview.org/content/implications-law-user-innovation]		While some focused on the potential consumer revolution or the composition of the crowd, others examined the business-related aspects of crowdsourcing. Identifying attributes of successful crowd innovators also has a business dimension. One researcher suggests that having experience spanning across a variety of communities or disciplines makes one likely to be considered 'innovative.'[http://74.125.155.132/scholar?q=cache:Ro0Ij3Y41wMJ:scholar.google.com/&hl=en&as_sdt=40000000&sciodt=40000000] Others focus more broadly on how to use the crowd to maintain or bolster business or brand. In ''Groundswell'' (2008),[http://books.google.com/books?id=YOVuQFXNcP4C&printsec=frontcover&dq=groundswell&hl=en&ei=ZF_tTKjtLIL58Ab-tu26AQ&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCsQ6wEwAA#v=onepage&q&f=false] Charlene Li and Josh Bernoff focus on how to most effectively use crowdsourcing to advantage businesses. The authors highlight how user bases of products can undermine a product or brand.[http://books.google.com/books?id=YOVuQFXNcP4C&printsec=frontcover&dq=groundswell&hl=en&src=bmrr&ei=tUPtTPOeAYP-8AbcxemdAQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCgQ6AEwAA#v=onepage&q&f=false] As a result, the authors propose that businesses use the "groundswell" to their advantage, fostering communities that can provide valuable feedback and economic payoffs. Marion K. Poetz and Martin Schreier have also taken a business perspective on crowdsourcing,[http://ssrn.com/abstract=1566903] arguing that the crowd is capable of producing valuable (but not always viable) business ideas at a low cost. Other researchers have found that young entrepreneurs who were attempting to start businesses frequently belonged to these kinds of communities.[http://ssrn.com/abstract=1022358] For a related discussion on user innovation and user communities, see Eric Von Hippel's books[http://web.mit.edu/evhippel/www/books.htm] and William Fisher's article in the Minnesota Law Review.[http://www.minnesotalawreview.org/content/implications-law-user-innovation]

5.68.47.58 at 22:26, 28 February 2019

2019-02-28T22:26:53Z

← Older revision		Revision as of 22:26, 28 February 2019
Line 43:		Line 43:
	1. Microtasks: ''Amazon's Mechanical Turk'', ''Microtask.com'', and ''Soylent''[https://www.mturk.com/mturk/welcome];		1. Microtasks: ''Amazon's Mechanical Turk'', ''Microtask.com'', and ''Soylent''[https://www.mturk.com/mturk/welcome];

	2. Tasks requiring "professional" skills: ''99designs''[http://99designs.com/] and ''InnoCentive''[http://www2.innocentive.com/]; ~~and ''Nerdy Writers''[https://www.nerdywriters.co.uk]~~		2. Tasks requiring "professional" skills: ''99designs''[http://99designs.com/] and ''InnoCentive''[http://www2.innocentive.com/];

	3. "Game" tasks: ''Gwap''[http://www.gwap.com/gwap/].		3. "Game" tasks: ''Gwap''[http://www.gwap.com/gwap/].

62.80.162.111 at 12:54, 10 December 2018

2018-12-10T12:54:43Z

← Older revision		Revision as of 12:54, 10 December 2018
Line 43:		Line 43:
	1. Microtasks: ''Amazon's Mechanical Turk'', ''Microtask.com'', and ''Soylent''[https://www.mturk.com/mturk/welcome];		1. Microtasks: ''Amazon's Mechanical Turk'', ''Microtask.com'', and ''Soylent''[https://www.mturk.com/mturk/welcome];

	2. Tasks requiring "professional" skills: ''99designs''[http://99designs.com/] and ''InnoCentive''[http://www2.innocentive.com/]; and		2. Tasks requiring "professional" skills: ''99designs''[http://99designs.com/] and ''InnoCentive''[http://www2.innocentive.com/]; and ''Nerdy Writers''[https://www.nerdywriters.co.uk]

	3. "Game" tasks: ''Gwap''[http://www.gwap.com/gwap/].		3. "Game" tasks: ''Gwap''[http://www.gwap.com/gwap/].

190.231.226.128: wFbB21 Current blog, fresh information, I read it from time to time!!...

2012-02-18T19:08:36Z

wFbB21 Current blog, fresh information, I read it from time to time!!...

← Older revision		Revision as of 19:08, 18 February 2012
Line 1:		Line 1:
	~~== Crowdsourcing: Background and Working Definitions ==~~		wFbB21 Current blog, fresh information, I read it from time to time!!...
	~~=== Definitions ===~~

	~~At present there is no generally accepted definition of crowdsourcing~~, and commentators have used many different meanings. Therefore, we believe an overview of the various definitions offered is helpful for further discussion of different types of crowdsourcing. For the purposes of our discussion, we will be using the terms "crowdsourcing", ~~"crowdwork", and "cloudwork" interchangeably.~~

	~~The most widely accepted definition of crowdsourcing comes from Jeff P. Howe, who recognized~~ it as "the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call."[http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html]

	~~He further clarified that the form of crowdsourcing could be either peer production (when co-workers interact and collaborate on projects) or sole individuals (when co-workers, if any, are isolated~~ from one another). Under Howe's definition, the employer must be an organization (in most cases, a corporation), because he was considering crowdsourcing as a new type of corporate business model, by which corporations could raise current productivity or establish new businesses that were not possible before. Nevertheless, we do not think the employer using crowdsourcing, as a matter of definition, must be an organization; individuals can certainly outsource a task to ~~an online crowd.~~

	Kleemann and Vob (2008) argue that "central to the concept of crowdsourcing is the idea that a crowd of people, collaboratively (or at least simultaneously) contribute to an aspect of the production process or to the solution of a design issue or other problems."[http://www.sti-studies.de/fileadmin/articles/kleemannetalstivol4no1.pdf]

	Although we agree that simultaneous or collaborative work is a significant type of crowdsourcing, it is not the only one. The Best Practices entry for crowdwork, developed last year and reposted on [[Class 3]], divides crowdwork into three categories: "First, a large group of workers may do microtasks to complete a whole project; the best-known platform in this arena is Amazon Mechanical Turk. Second, companies may use cloudwork platforms to connect with individual workers, or a small group of workers, who then complete larger jobs (e.g., Elance[http://www.elance.com/p/landing/buyerA5.html] and oDesk[http://www.odesk.com/#reloaded]). Finally, a company may run 'contests,' where numerous workers complete a task and only the speediest or best worker is paid (e.g., InnoCentive[https://www2.innocentive.com/] and Worth1000[http://www.worth1000.com/]). In some contests, the company commits to picking at least one winner; in others, there is no such guarantee." It is clear that when the crowdsourcing takes the form of competitive bidding, not every participant works on a single aspect of the task; each of them works on the whole task, and they do not have to work at the same time. ~~Only the final winner gets compensated. It is possible that only one individual or organization joins the bidding process and no competing parties are involved.~~

	Another concept that Reichwald and Piller (2006) used to describe crowdsourcing is "interactive value creation". They further differentiate two types of crowdsourcing: mass customization and open innovation.[http://www.sti-studies.de/fileadmin/articles/kleemannetalstivol4no1.pdf] We do not think Reichwald and Piller's approach is convincing. First, the general term "interactive value creation" can be used to cover many types of online collaborative activities traditionally not recognized as crowdsourcing, e.g., open source development. Second, mass customization refers to an isolated customer's activity to tailor one particular product, rather than contributions to a general product. Their second type, "open innovation", correctly points out that crowdsourcing should be outsourced through a open call, but we do not believe a strong degree of "innovation" is necessary, especially for highly divided microtasks.

	Based on the above discussion, we believe that there are two core elements of crowdsourcing, both of which may be facilitated by an online platform (such as Amazon Mechanical Turk[https://www.mturk.com/mturk/welcome])

	~~1. The task is outsourced through an open call from the employer;~~

	~~2. The recipients of the call, whether or not they elect to participate, comprise a large, amorphous crowd.~~

	The following discussion of crowdsourcing reflects our understanding of three of the most significant types of crowdsourcing: microtasks, "professional" tasks, and "game" tasks. Microtasks facilitated by platforms such as Amazon Mechanical Turk [https://www.mturk.com/mturk/welcome] might be the typical type of crowdsourcing in literature.[http://www.clickadvisor.com/downloads/Howe_The_Rise_of_Crowdsourcing.pdf] Not only is there an open call by an employer to the crowd, the crowd also collaborates on the whole task by each member addressing a small piece of the pre-divided work. Professional tasks, in contrast, might have multiple people working on them, but there is no collaboration among them, and their work usually involve a higher degree of innovation. Game tasks are again a different animal, since they copy many attributes of the other two, but add another layer by aiming to be entertaining as well as purposeful.

	~~=== General Information on Crowdsourcing ===~~
	*''General Information''
	**For a quick overview by Jeff Howe, author of ''Crowdsourcing'',[http://books.google.com/books?id=ge_0LBOcwWsC&printsec=frontcover&dq=crowdsourcing&hl=en&ei=QinGTKS9AcGAlAeHtLX-AQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CC8Q6AEwAA#v=onepage&q&f=false] take a look at this YouTube clip.[http://www.youtube.com/watch?v=F0-UtNg3ots]
	**Northwestern University Professor Kris Hammond also explains crowdsourcing, but argues that its downsides are worker rewards and quality.[http://www.youtube.com/watch?v=eX7RiV-wa_s&feature=related]
	**Our very own Jonathan Zittrain discusses crowdsourcing in his talk, ''Minds for Sale''.[http://www.youtube.com/watch?v=Dw3h-rae3uo]
	**Several individuals gathered to discuss crowdsourcing in a panel moderated by New York Times correspondent Brad Stone.[http://www.youtube.com/watch?v=lxyUaWSblaA]
	*''In the News''
	**The New York Times recently ran an article on crowdsourcing featuring two crowdsourcing companies:[http://www.nytimes.com/2010/10/31/business/31digi.html?_r=1&ref=technology] Microtask[http://www.microtask.com/] and CloudCrowd.[http://www.cloudcrowd.com/]
	**It's interesting to note that these companies are attempting to monetize crowdsourcing in exactly the way in which Howe says it cannot be monetized successfully.
	*''Examples of Crowdsourcing''
	**Take a look at Wikipedia's compilation.[http://en.~~wikipedia~~.~~org/wiki/List_of_crowdsourcing_projects]~~

	== Crowdsourcing Literature ==		== Crowdsourcing Literature ==

Rachelbfunk: /* Compensation */

2010-12-12T16:52:15Z

Compensation

← Older revision		Revision as of 16:52, 12 December 2010
Line 253:		Line 253:
	One might argue that the issue is not that companies would be penalized were they forced to compensate the players for playing games with a purpose, but rather that the workers are being currently penalized through the lack of compensation because they are unwittingly contributing free labor when they are entitled to a fair wage for their work. The question then becomes whether playing such games is indeed "work." We would argue that it is not, because people are playing the game primarily for its entertainment value: if it stops being entertaining, then they will stop playing it. Alternatively, if they are playing it because they support the underlying purpose, then that purpose is clearly important enough to them that they are doing it for free, so they already have sufficient incentive to do the work. After all, do we want to encourage a value system where everything must be set at a price? (If the answer to that question is yes, Wikipedia is in trouble.)		One might argue that the issue is not that companies would be penalized were they forced to compensate the players for playing games with a purpose, but rather that the workers are being currently penalized through the lack of compensation because they are unwittingly contributing free labor when they are entitled to a fair wage for their work. The question then becomes whether playing such games is indeed "work." We would argue that it is not, because people are playing the game primarily for its entertainment value: if it stops being entertaining, then they will stop playing it. Alternatively, if they are playing it because they support the underlying purpose, then that purpose is clearly important enough to them that they are doing it for free, so they already have sufficient incentive to do the work. After all, do we want to encourage a value system where everything must be set at a price? (If the answer to that question is yes, Wikipedia is in trouble.)

	Moreover, it should also be noted that not all workers get paid; some are volunteers. For instance, in ''Crowdsourcing'', Jeff Howe talks about the contribution of the crowd to tasks like NASA’s "Clickworkers" project (2006:62)[https://secure.wikimedia.org/wikipedia/en/wiki/Clickworkers] and the Cornell Lab of Ornithology’s eBird project (2006:31)[http://ebird.org/content/ebird/about]. In both examples, the crowd knows that what they are doing is work, but they still do it, even though they are contributing information for free, because they support the purpose for which they are laboring. If anything, this emphasizes the importance of disclosure rather than compensation.		Moreover, it should also be noted that not all workers get paid; some are volunteers. For instance, in ''Crowdsourcing'', Jeff Howe talks about the contribution of the crowd to tasks like NASA’s "Clickworkers" project (2006: 62)[https://secure.wikimedia.org/wikipedia/en/wiki/Clickworkers] and the Cornell Lab of Ornithology’s eBird project (2006: 31)[http://ebird.org/content/ebird/about]. In both examples, the crowd knows that what they are doing is work, but they still do it, even though they are contributing information for free, because they support the purpose for which they are laboring. If anything, this emphasizes the importance of disclosure rather than compensation.

	== Summary ==		== Summary ==

	Each of the three types of crowdsourcing discussed above presents interesting challenges based on the unique characteristics that each possesses. While there is some overlap among them -- for example, the ESP Game is essentially a microtask designed as a game -- each manifests its "difficult problems" in a different way. For instance, while compensation is a potential issue for all three, there is a concern present with microtasks that workers are not sufficiently compensated to make a living wage; unease that higher education is being devalued through lower wages for "professional" tasks; and a question whether workers are somehow being duped into doing work for free with crowdsourcing games. While the Best Practices document outlines great general solutions to problems, we felt that the different types of crowdsourcing required more tailored responses. Thus, we have attempted here to offer specific answers to some of the problems within each kind of crowdsourcing, while also providing a look at crowdsourcing generally and the disputes currently taking place in the literature. The next step from here might be to do a series of demonstration projects, utilizing and testing some of the suggestions here.		Each of the three types of crowdsourcing discussed above presents interesting challenges based on the unique characteristics that each possesses. While there is some overlap among them -- for example, the ESP Game is essentially a microtask designed as a game -- each manifests its "difficult problems" in a different way. For instance, while compensation is a potential issue for all three, there is a concern present with microtasks that workers are not sufficiently compensated to make a living wage; unease that higher education is being devalued through lower wages for "professional" tasks; and a question whether workers are somehow being duped into doing work for free with crowdsourcing games. While the Best Practices document outlines great general solutions to problems, we felt that the different types of crowdsourcing required more tailored responses. Thus, we have attempted here to offer specific answers to some of the problems within each kind of crowdsourcing, while also providing a look at crowdsourcing generally and the disputes currently taking place in the literature. The next step from here might be to do a series of demonstration projects, utilizing and testing some of the suggestions here.

Rachelbfunk: /* Reputation */

2010-12-12T16:48:15Z

Reputation

← Older revision		Revision as of 16:48, 12 December 2010
Line 166:		Line 166:
	''Reputation Concerns for Workers, Employers, and Platforms''		''Reputation Concerns for Workers, Employers, and Platforms''

	* Workers. As crowdsourcing increases and professionalized crowdworkers have success, crowdworkers want to signal to potential employers that they have done good work in the past. Workers may also want to list their professional experience, education, or training (as one can do on Gerson Lehram Group[http://www.glgroup.com/]). Workers also will seek to ensure that antisocial behavior among themselves is counted against a worker's reputation. The literature on crowdsourcing emphasizes the community that exists in crowdwork environment, and ensuring the community functions well is important. 99designs has numerous instances where a crowdworker accuses another of "stealing" ~~another~~'s design. Workers also will want to know the reputation of an employer. This could include factors like the amount paid, the quickness of payment, and the type of work offered.		* Workers. As crowdsourcing increases and professionalized crowdworkers have success, crowdworkers want to signal to potential employers that they have done good work in the past. Workers may also want to list their professional experience, education, or training (as one can do on Gerson Lehram Group[http://www.glgroup.com/]). Workers also will seek to ensure that antisocial behavior among themselves is counted against a worker's reputation. The literature on crowdsourcing emphasizes the community that exists in crowdwork environment, and ensuring the community functions well is important. 99designs has numerous instances where a crowdworker accuses another of "stealing" someone's design. Workers also will want to know the reputation of an employer. This could include factors like the amount paid, the quickness of payment, and the type of work offered.

	* Employers. Reputation also ~~would~~ be valuable for employers--they'd prefer work from people with high reputations, not only to ensure good work, but also to ensure the work was not copied from somewhere else. While copying from the worker's perspective is important as a social matter, from an employer's perspective copying is important as a legal matter. Employers want to avoid copyright infringement or any claims of copyright infringement. Employers also want to broadcast their "good" reputations.		* Employers. Reputation would also be valuable for employers--they'd prefer work from people with high reputations, not only to ensure good work, but also to ensure the work was not copied from somewhere else. While copying from the worker's perspective is important as a social matter, from an employer's perspective copying is important as a legal matter. Employers want to avoid copyright infringement or any claims of copyright infringement. Employers also want to broadcast their "good" reputations.

	* Platforms. Platforms have an incentive to ensure that workers' and employers' desire for a reputation system is met. It will, in many cases, be up to the platform to institute a system that can deal effectively with reputation, ~~antiscoial~~ behavior, and legal issues. 99designs has implemented a system that seeks to deal with the latter two, but does not address reputation.		* Platforms. Platforms have an incentive to ensure that workers' and employers' desire for a reputation system is met. It will, in many cases, be up to the platform to institute a system that can deal effectively with reputation, antisocial behavior, and legal issues. 99designs has implemented a system that seeks to deal with the latter two, but does not address reputation.


Line 185:		Line 185:
	*2. Uniform Reputation. Another approach to reputation would use a centralized entity to manage worker/employer reputation across all platforms. One can imagine a universal reputation platform that draws information from various sites and aggregates it--maybe it has agreements with all the platforms to provide it reputation-related information. (There are already sites that purport to provide reputational rankings of websites--see, e.g., Webutation.[http://www.webutation.net/#]) In other words, this entity would maintain a database of all crowdsourced employers/workers reputations. Reputational information would be aggregated automatically from all participating crowdsourcing platforms, which would automatically submit data to this entity. Theoretically, employers and workers could consult this platform to determine worker reputations. To enable this "reputation finding," the reputation database could contain various "categories" of workers and employees, and allow an individual to sort based on various reputational scores, which would be determined based on a variety of metrics and the information provided to the centralized entity by the crowdsourcing platforms. So, for example, a company could find someone with a high "creative reputation" (e.g., logo design) or someone with a high "engineering" rating. It may also allow users to disaggregate reputational information across platforms to see how the users has performed/behaved in each. Additionally, a central reputation-authority would allow people to move "with" their reputations and stay "anonymous"--they won't disclose their real identity. As noted, this all would require platforms to (automatically) submit information to this reputation database/entity. Because platforms (in the hypothetical world) would automatically submit worker reputation info to the site, there are essentially two ways the site could work at the level of individual reputations. Both methods of operation work from the premise that individuals and employers are searchable by reputation. Search results might display an aggregate reputation or a simple ranking with no reputational score accompanying it. The first way resembles the solution described above: once a user is selected, a visitor is taken to a reputation homepage that displays a worker/employer reputation from different platforms, and provides information about each. This provides at least one benefit for platforms: they don't have to generate interoperability--they can delegate reputation aggregation to one entity, which could probably collect and display the information more effectively. Two authors have already sketched a design for a similar system.[http://sandeepkumar.org/my/papers/2009_CAT_PortableReputation.pdf] A second method would be to for the reputation site to create its own "general" reputation for each worker based on the information it receives from platforms. We can imagine a situation in which workers or employers can ask the reputation site to rank workers based on specific, general, or a combination of characteristics. This would allow employers to sort workers by reputational characteristics they deem important to a particular type of work. Once sorted, employers could provide an open-call project to the narrowed crowd of workers. This system also would require the centralized agency to weight different sites or reputational attributes. This is a problem of reliability, discussed below. One other advantage of a unified reputational system would be allowing some user control over reputation. We can imagine a place in a user profile where the user can fill in information, provide explanations, or otherwise comment on past work experience and the like. (There may also be room for a centralized-decentralized approach, where the user claims authority over her reputation and distributes it how she wants. Jon Udell outlined such a concept in his recent talk at the Berkman Center.[http://cyber.law.harvard.edu/events/luncheon/2010/12/udell])		*2. Uniform Reputation. Another approach to reputation would use a centralized entity to manage worker/employer reputation across all platforms. One can imagine a universal reputation platform that draws information from various sites and aggregates it--maybe it has agreements with all the platforms to provide it reputation-related information. (There are already sites that purport to provide reputational rankings of websites--see, e.g., Webutation.[http://www.webutation.net/#]) In other words, this entity would maintain a database of all crowdsourced employers/workers reputations. Reputational information would be aggregated automatically from all participating crowdsourcing platforms, which would automatically submit data to this entity. Theoretically, employers and workers could consult this platform to determine worker reputations. To enable this "reputation finding," the reputation database could contain various "categories" of workers and employees, and allow an individual to sort based on various reputational scores, which would be determined based on a variety of metrics and the information provided to the centralized entity by the crowdsourcing platforms. So, for example, a company could find someone with a high "creative reputation" (e.g., logo design) or someone with a high "engineering" rating. It may also allow users to disaggregate reputational information across platforms to see how the users has performed/behaved in each. Additionally, a central reputation-authority would allow people to move "with" their reputations and stay "anonymous"--they won't disclose their real identity. As noted, this all would require platforms to (automatically) submit information to this reputation database/entity. Because platforms (in the hypothetical world) would automatically submit worker reputation info to the site, there are essentially two ways the site could work at the level of individual reputations. Both methods of operation work from the premise that individuals and employers are searchable by reputation. Search results might display an aggregate reputation or a simple ranking with no reputational score accompanying it. The first way resembles the solution described above: once a user is selected, a visitor is taken to a reputation homepage that displays a worker/employer reputation from different platforms, and provides information about each. This provides at least one benefit for platforms: they don't have to generate interoperability--they can delegate reputation aggregation to one entity, which could probably collect and display the information more effectively. Two authors have already sketched a design for a similar system.[http://sandeepkumar.org/my/papers/2009_CAT_PortableReputation.pdf] A second method would be to for the reputation site to create its own "general" reputation for each worker based on the information it receives from platforms. We can imagine a situation in which workers or employers can ask the reputation site to rank workers based on specific, general, or a combination of characteristics. This would allow employers to sort workers by reputational characteristics they deem important to a particular type of work. Once sorted, employers could provide an open-call project to the narrowed crowd of workers. This system also would require the centralized agency to weight different sites or reputational attributes. This is a problem of reliability, discussed below. One other advantage of a unified reputational system would be allowing some user control over reputation. We can imagine a place in a user profile where the user can fill in information, provide explanations, or otherwise comment on past work experience and the like. (There may also be room for a centralized-decentralized approach, where the user claims authority over her reputation and distributes it how she wants. Jon Udell outlined such a concept in his recent talk at the Berkman Center.[http://cyber.law.harvard.edu/events/luncheon/2010/12/udell])

	*3. Reliability for Platform-Specific Versus Uniform. Each system of reputation (platform specific and uniform) will have issues of reliability. By reliability we mean a platform can accurately report to a user how well or poorly a specific individual has performed in the past, where performance is based specific criteria. Because reputation systems rely on information, what information is used and who discloses it can influence the reliability of a reputation system.[http://www.unpcdc.org/media/6406/reputation%20mechanisms%20and%20electronic%20markets%20-%20economic%20issues%20and%20proposals%20for%20public%20procurement.pdf]		*3. Reliability for Platform-Specific Versus Uniform. Each system of reputation (platform specific and uniform) will have issues of reliability. By reliability, we mean a platform can accurately report to a user how well or poorly a specific individual has performed in the past, where performance is based specific criteria. Because reputation systems rely on information, what information is used and who discloses it can influence the reliability of a reputation system.[http://www.unpcdc.org/media/6406/reputation%20mechanisms%20and%20electronic%20markets%20-%20economic%20issues%20and%20proposals%20for%20public%20procurement.pdf]
	**Platform-specific reputational systems would likely achieve greater reliability because they would be localized. As self-contained systems, each platform could respond to user demands, as well as tinker with existing formulae that garner ratings. Still, it's not clear how reliable these local reputational systems could be across platforms. Assuming for the moment that each platform can in some way support the reputational rating of another platform, we still might have the problem of "overload": individuals will have ''too many'' reputational scores from ''too many'' different platforms. Employers may then look or sort only to the lowest rating of any given platform on a user profile. Thus, platform specific rating systems could actually work to the detriment of the users.		**Platform-specific reputational systems would likely achieve greater reliability because they would be localized. As self-contained systems, each platform could respond to user demands, as well as tinker with existing formulae that garner ratings. Still, it's not clear how reliable these local reputational systems could be across platforms. Assuming for the moment that each platform can in some way support the reputational rating of another platform, we still might have the problem of "overload": individuals will have ''too many'' reputational scores from ''too many'' different platforms. Employers may then look or sort only to the lowest rating of any given platform on a user profile. Thus, platform specific rating systems could actually work to the detriment of the users.
	**A uniform reputational system may face more reliability issues than an interoperable platform-specific reputational system. For one thing, a unified system will be drawing reputation data from hundreds of different platforms automatically. There is a risk for gaming the system on two levels here. First, a user at a relatively unsophisticated platform may try to manipulate the system and boost their own reputational rating. Second, the platform itself may have an incentive to give its users higher reputational scores. Why? Because assuming the unified system computers some aggregate reputational score, any platform can boost their users' scores and give them an advantage in the market place. Another problem arises as to weighting reputational scores. Should a reputation score from InnoCentive be "worth" more than one from Gerson Lehrman? Do reputation scores from Threadless even translate into anything meaningful? If so, should they be treated the same as scores from 99designs. All of these questions illustrate that there is no easy way to decide how scores should be weighted, though we can think of a variety of factors, including number of users, length of existence, customer satisfaction, etc. Part of the problem also is that different platforms may use different rating systems--some may use only one general rating, while others may parse ratings into various categories. How these issues are dealt with impacts the "reliability" of reputation. It also, therefore, impacts the trust that crowdworkers have in a reputational system.		**A uniform reputational system may face more reliability issues than an interoperable platform-specific reputational system. For one thing, a unified system will be drawing reputation data from hundreds of different platforms automatically. There is a risk for gaming the system on two levels here. First, a user at a relatively unsophisticated platform may try to manipulate the system and boost their own reputational rating. Second, the platform itself may have an incentive to give its users higher reputational scores. Why? Because assuming the unified system computers some aggregate reputational score, any platform can boost their users' scores and give them an advantage in the market place. Another problem arises as to weighting reputational scores. Should a reputation score from InnoCentive be "worth" more than one from Gerson Lehrman? Do reputation scores from Threadless even translate into anything meaningful? If so, should they be treated the same as scores from 99designs. All of these questions illustrate that there is no easy way to decide how scores should be weighted, though we can think of a variety of factors, including number of users, length of existence, customer satisfaction, etc. Part of the problem also is that different platforms may use different rating systems--some may use only one general rating, while others may parse ratings into various categories. How these issues are dealt with impacts the "reliability" of reputation. It also, therefore, impacts the trust that crowdworkers have in a reputational system.

Rachelbfunk: /* "Deprofessionalization" */

2010-12-12T16:46:30Z

"Deprofessionalization"

← Older revision		Revision as of 16:46, 12 December 2010
Line 142:		Line 142:
	''Specific Problems''		''Specific Problems''

	*1. Cannibalization/Wage Reduction. If 99designs or InnoCentive lowers entry barriers and costs, it could stimulate a race to the bottom. 99designs' Business Development and Marketing Manager, Jason Aiken, has already ~~acknolwedged~~ that the company has created "a tension" with the traditional market for graphic design--and it may be driving down wages.[http://www.youtube.com/watch?v=lxyUaWSblaA] (Clip at 34:16.) In this world, professionals could not earn a living because "amateurs" or other professionals that do not have jobs will drive down prices for crowdsourced design work. With low prices and an abundance of crowdworkers, companies may shed their traditional means of acquiring designs. So crowdsourcing, which started off as a way to lower specific business costs or solve thorny problems, becomes the sole means of (research &) design work. In this environment, which designers worry about,[http://www.xemion.com/blog/99designscom-a-warning-to-freelancers-67.html] the professional industry collapses because wages are too low. Alternatively, a new web-based professionalization occurs. That, of course, depends on a variety of factors, including the ability of crowdworkers to maintain a coherent identity/reputation online (see Reputation below).		*1. Cannibalization/Wage Reduction. If 99designs or InnoCentive lowers entry barriers and costs, it could stimulate a race to the bottom. 99designs' Business Development and Marketing Manager, Jason Aiken, has already acknowledged that the company has created "a tension" with the traditional market for graphic design--and it may be driving down wages.[http://www.youtube.com/watch?v=lxyUaWSblaA] (Clip at 34:16.) In this world, professionals could not earn a living because "amateurs" or other professionals that do not have jobs will drive down prices for crowdsourced design work. With low prices and an abundance of crowdworkers, companies may shed their traditional means of acquiring designs. So crowdsourcing, which started off as a way to lower specific business costs or solve thorny problems, becomes the sole means of (research &) design work. In this environment, which designers worry about,[http://www.xemion.com/blog/99designscom-a-warning-to-freelancers-67.html] the professional industry collapses because wages are too low. Alternatively, a new web-based professionalization occurs. That, of course, depends on a variety of factors, including the ability of crowdworkers to maintain a coherent identity/reputation online (see Reputation below).

	*2. Devaluing of Education. If deprofessionalization occurs and an industry cannibalizes itself, there will be a concomitant and precipitous drop in the market value of education. In scientific areas, many crowdworkers tend to have advanced degrees, or are highly educated. Their ability to perform sophisticated or professional crowdwork is therefore partly dependent upon their education. But the crowdwork market undervalues the educational experience of the worker because, at present, most highly-educated workers are engaged in part-time, non-sustenance activity. As crowdwork becomes more popular for professionals, wages fall and crowdwork replaces traditional professional work. The cost of education, however, remains constant. This means that sophisticated crowdworkers will pay the same amount for school but will be full-time, instead of part-time, crowdworkers. Given the low wages, people will not be making an adequate return on their educational investment. Several possibilities then result--but we focus on the most dire here. Knowing their inability to generate an adequate return on their investment, low wages could deter individuals from higher education. This, in turn, will cause a brain drain, where fewer and fewer people obtain advanced degrees. As a result, the market for professional crowdworkers shrinks. Given the technological growth rate, the demand for crowdworkers will continue to grow. The shorty supply and high prices will mean the end of crowdsourced professional work for two reasons. Costs will rise to the point where crowdsourcing is no longer more economical than traditional professional services. Second, and more importantly, the lack of workers means that, given the growing technological and cultural demands our society makes, tasks simply cannot be crowdsourced effectively. Although one might argue that markets will correct for the supply/demand problems, it may not do so at fast enough rates. The legal education market, for example, has continually grown despite decreasing demand.[http://articles.chicagotribune.com/2010-04-27/business/ct-biz-0427-chicago-law-students--20100427_1_law-school-law-firms-national-law-journal] The same thing is happening with for-profit colleges.[http://www.pbs.org/wgbh/pages/frontline/collegeinc/][http://chronicle.com/article/For-Profit-Colleges-Mount-a/66027/] The market has not yet corrected itself because the government is so willing to issue loans. When the problem finally comes to the fore, there will be serious costs, both to society, to students, to graduates, and to the industry. The mere fact that the market will correct the problem eventually doesn't mean we should wait for it to do so, especially when the correction may be painful and enduring.		*2. Devaluing of Education. If deprofessionalization occurs and an industry cannibalizes itself, there will be a concomitant and precipitous drop in the market value of education. In scientific areas, many crowdworkers tend to have advanced degrees, or are highly educated. Their ability to perform sophisticated or professional crowdwork is therefore partly dependent upon their education. But the crowdwork market undervalues the educational experience of the worker because, at present, most highly-educated workers are engaged in part-time, non-sustenance activity. As crowdwork becomes more popular for professionals, wages fall and crowdwork replaces traditional professional work. The cost of education, however, remains constant. This means that sophisticated crowdworkers will pay the same amount for school but will be full-time, instead of part-time, crowdworkers. Given the low wages, people will not be making an adequate return on their educational investment. Several possibilities then result--but we focus on the most dire here. Knowing their inability to generate an adequate return on their investment, low wages could deter individuals from higher education. This, in turn, will cause a brain drain, where fewer and fewer people obtain advanced degrees. As a result, the market for professional crowdworkers shrinks. Given the technological growth rate, the demand for crowdworkers will continue to grow. The shorty supply and high prices will mean the end of crowdsourced professional work for two reasons. Costs will rise to the point where crowdsourcing is no longer more economical than traditional professional services. Second, and more importantly, the lack of workers means that, given the growing technological and cultural demands our society makes, tasks simply cannot be crowdsourced effectively. Although one might argue that markets will correct for the supply/demand problems, it may not do so at fast enough rates. The legal education market, for example, has continually grown despite decreasing demand.[http://articles.chicagotribune.com/2010-04-27/business/ct-biz-0427-chicago-law-students--20100427_1_law-school-law-firms-national-law-journal] The same thing is happening with for-profit colleges.[http://www.pbs.org/wgbh/pages/frontline/collegeinc/][http://chronicle.com/article/For-Profit-Colleges-Mount-a/66027/] The market has not yet corrected itself because the government is so willing to issue loans. When the problem finally comes to the fore, there will be serious costs, both to society, to students, to graduates, and to the industry. The mere fact that the market will correct the problem eventually doesn't mean we should wait for it to do so, especially when the correction may be painful and enduring.
Line 148:		Line 148:
	''Solutions.''		''Solutions.''

	One might wonder whether concerns over deprofessionalization are overstated. The criticism--the one Howe addresses the first chapter ("The Rise of the Amateur") in his book[http://books.google.com/books?id=LRbsMBxR9ykC&printsec=frontcover&dq=howe+crowdsourcing&hl=en&ei=xj_4TMLDIcGB8ga3w5mLAg&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCsQ6wEwAA#v=onepage&q&f=false]--is that we are worried simply about "amateurs" displacing "professionals." Howe certainly has a point: if other individuals can do the work of professionals at less wages and through crowdsourcing, then what are we worried about? We do, of course, like things to be efficient. We are concerned, however, that some industries could, over the long run, suffer damage if efficiency is our only ~~conern~~. That doesn't mean that industries will necessarily suffer damage; but they may. And we feel it's worth considering what could happen if we place all our bets on markets and efficiency, and the dealer comes up 21. One place where we are starting to see problems of relying on market is with traditional media outlets. We may think that "amateurs" can report just as well as many professionals--and their doing so may be more efficient. But we have other concerns, such as whether the reporting is reliable and whether the ~~fact~~. Similarly, in design work, efficiency and opportunity are good things. But what happens when the market for professionals has disappeared, and only low-wage crowdworkers remain? Alternatively, what if all legal work ~~was~~ crowdsourced? The quality of work may decrease, relationships between employers and workers may fray, and the industry may have significant difficulties. As noted, though, this is not the only scenario. The industry may adapt to these changes and do just fine. The exercise here, though, is to think about deprofessionalization and education devaluation as problems, and pose potential solutions while recognizing they may not materialize. Here are a few solutions we might consider:		One might wonder whether concerns over deprofessionalization are overstated. The criticism--the one Howe addresses the first chapter ("The Rise of the Amateur") in his book[http://books.google.com/books?id=LRbsMBxR9ykC&printsec=frontcover&dq=howe+crowdsourcing&hl=en&ei=xj_4TMLDIcGB8ga3w5mLAg&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCsQ6wEwAA#v=onepage&q&f=false]--is that we are worried simply about "amateurs" displacing "professionals." Howe certainly has a point: if other individuals can do the work of professionals at less wages and through crowdsourcing, then what are we worried about? We do, of course, like things to be efficient. We are concerned, however, that some industries could, over the long run, suffer damage if efficiency is our only concern. That doesn't mean that industries will necessarily suffer damage, but they may. And we feel that it's worth considering what could happen if we place all our bets on markets and efficiency, and the dealer comes up 21. One place where we are starting to see problems of relying on market is with traditional media outlets. We may think that "amateurs" can report just as well as many professionals--and their doing so may be more efficient. But we have other concerns, such as whether the reporting is reliable and whether the facts check out. Similarly, in design work, efficiency and opportunity are good things. But what happens when the market for professionals has disappeared, and only low-wage crowdworkers remain? Alternatively, what if all legal work were crowdsourced? The quality of work may decrease, relationships between employers and workers may fray, and the industry may have significant difficulties. As noted, though, this is not the only scenario. The industry may adapt to these changes and do just fine. The exercise here, though, is to think about deprofessionalization and education devaluation as problems, and pose potential solutions while recognizing they may not materialize. Here are a few solutions we might consider:

	*1. Wage Scale. Industries and professionals could collaborate to set wages they think are reasonable. The Best Practices document recommends "fair" wages, but seems to presume only employers will be the ones deciding what wages to set. In this professional-crowdwork context, it might be wise to allow collaboration of interested parties, rather than rely only on employers to set a fair wage. One could see such a wage scale being set by various stakeholders, and perhaps some "non-stakeholders."		*1. Wage Scale. Industries and professionals could collaborate to set wages they think are reasonable. The Best Practices document recommends "fair" wages, but seems to presume that only employers will be the ones deciding what wages to set. In this professional-crowdwork context, it might be wise to allow collaboration of interested parties, rather than rely only on employers to set a fair wage. One could see such a wage scale being set by various stakeholders, and perhaps some "non-stakeholders."
	*2. Wage Determinants. Wages could be set according to some or a series credential-measures. This could take several ~~different~~ forms, some of which could be combined.		*2. Wage Determinants. Wages could be set according to some or a series credential-measures. This could take several forms, some of which could be combined.
	**2a. Workers with a degree or work experience in a related professional field may be entitled to a higher wage than a worker with no higher education. The problem with this approach is that it reduces the "meritocracy" aspect of crowdsourcing. It also seems to devalue the diversity that crowdsourcing thrives on.		**2a. Workers with a degree or work experience in a related professional field may be entitled to a higher wage than a worker with no higher education. The problem with this approach is that it reduces the "meritocracy" aspect of crowdsourcing. It also seems to devalue the diversity that crowdsourcing thrives on.
	**2b. If the platform implements an effective reputation or rating system (detailed or simple), workers could use their reputations to generate more work. This solution faces some technical and privacy problems, but seems like at least one plausible way of ensuring favorable wages for better performers. This also has the benefit of showing which workers are repeatedly good at performing tasks. This is important because many InnoCentive solvers, for example, never solve more than one problem.[http://www.hbs.edu/research/pdf/07-050.pdf] One drawback of this method is that it decreases the "perfect meritocracy" that some seem to think can persist forever (if it exists at all).		**2b. If the platform implements an effective reputation or rating system (detailed or simple), workers could use their reputations to generate more work. This solution faces some technical and privacy problems, but seems like at least one plausible way of ensuring favorable wages for better performers. This also has the benefit of showing which workers are repeatedly good at performing tasks. This is important because many InnoCentive solvers, for example, never solve more than one problem.[http://www.hbs.edu/research/pdf/07-050.pdf] One drawback of this method is that it decreases the "perfect meritocracy" that some seem to think can persist forever (if it exists at all).
Line 158:		Line 158:
	*4. Discourage wage reductions/crowdsourcing by having platforms require disclosure when crowdsourcing. This, however, may discourage crowdsourcing generally. The goal should be to reap crowdsourcing's benefits while minimizing its potential downside. Still, this solution could be used to less extreme degrees to pressure companies to offer competitive wages for crowdsourced tasks.		*4. Discourage wage reductions/crowdsourcing by having platforms require disclosure when crowdsourcing. This, however, may discourage crowdsourcing generally. The goal should be to reap crowdsourcing's benefits while minimizing its potential downside. Still, this solution could be used to less extreme degrees to pressure companies to offer competitive wages for crowdsourced tasks.
	*5. Incentivize pay-friendly behavior by using reputation system (see below). The various approaches here could be combined with or pushed into a reputation system. We explain various problems with reputation systems below. Here, we focus on the potential benefits of a three-way reputation system. "Three-way" means that the system involves all parties: workers, employers, and platforms. A reputation that engages all three parties can incentivize each party to engage in wage-friendly behavior. A reputation system, for example, that allows workers or platforms to assign reputational rankings to employers can cause wages to increase. If workers know that Corporation A generally pays 2x as much as Corporation B, workers can identify "fairer" wages. Similarly, companies may be willing to offer better wages if they can screen out "low reputation" workers.		*5. Incentivize pay-friendly behavior by using reputation system (see below). The various approaches here could be combined with or pushed into a reputation system. We explain various problems with reputation systems below. Here, we focus on the potential benefits of a three-way reputation system. "Three-way" means that the system involves all parties: workers, employers, and platforms. A reputation that engages all three parties can incentivize each party to engage in wage-friendly behavior. A reputation system, for example, that allows workers or platforms to assign reputational rankings to employers can cause wages to increase. If workers know that Corporation A generally pays 2x as much as Corporation B, workers can identify "fairer" wages. Similarly, companies may be willing to offer better wages if they can screen out "low reputation" workers.
	*6. Regulation. One of the benefits of crowdsourcing is the lack of regulation. That is sure to change, and it might be that regulating ~~crowdsoucring~~, either as to wages or work, could benefit everyone.		*6. Regulation. One of the benefits of crowdsourcing is the lack of regulation. That is sure to change, and it might be that regulating crowdsourcing, either as to wages or work, could benefit everyone.

	===== Reputation =====		===== Reputation =====

← Older revision		Revision as of 15:41, 6 June 2020
Line 9:		Line 9:
	These books have provoked an academic interest in finding out ''who'' is the crowd, or why the crowd moves the way it does. Some have looked at scientific crowdsourcing, asking what characteristics make someone a successful crowdworker/problem-solver.[http://www.hbs.edu/research/pdf/07-050.pdf] Part of answering that question, it turns out, requires asking why people attempt to be part of the innovating crowd in the first place. The authors of this study found that the crowd was highly educated. They also found heterogeneity in scientific interests, as well as monetary and intrinsic motivations, to be important drivers of "good" problem-solvers [https://academicsciences.co.uk/about-us/]. Others have examined non-scientific endeavors and asked similar questions. One report, for example, examined workers on iStockphoto.[http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2159/1969] This report found that most iStock crowdworkers developing photographs were highly educated and motivated primarily by money. Jeff Howe, however, takes a different perspective as to crowdworkers' motivation: "There are . . . two shared attributes among almost all crowdsourcing projects: participants are not primarily motivated by money, and they’re donating their leisure hours to the cause. That is, they’re contributing their excess capacity, or 'spare cycles,' to indulge in something they love to do." (''Crowdsourcing'', pgs. 28-29.)		These books have provoked an academic interest in finding out ''who'' is the crowd, or why the crowd moves the way it does. Some have looked at scientific crowdsourcing, asking what characteristics make someone a successful crowdworker/problem-solver.[http://www.hbs.edu/research/pdf/07-050.pdf] Part of answering that question, it turns out, requires asking why people attempt to be part of the innovating crowd in the first place. The authors of this study found that the crowd was highly educated. They also found heterogeneity in scientific interests, as well as monetary and intrinsic motivations, to be important drivers of "good" problem-solvers [https://academicsciences.co.uk/about-us/]. Others have examined non-scientific endeavors and asked similar questions. One report, for example, examined workers on iStockphoto.[http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2159/1969] This report found that most iStock crowdworkers developing photographs were highly educated and motivated primarily by money. Jeff Howe, however, takes a different perspective as to crowdworkers' motivation: "There are . . . two shared attributes among almost all crowdsourcing projects: participants are not primarily motivated by money, and they’re donating their leisure hours to the cause. That is, they’re contributing their excess capacity, or 'spare cycles,' to indulge in something they love to do." (''Crowdsourcing'', pgs. 28-29.)

	While some focused on the potential consumer revolution or the composition of the crowd, others examined the business-related aspects of crowdsourcing. Identifying attributes of successful crowd innovators also has a business dimension. One researcher suggests that having experience spanning across a variety of communities or disciplines makes one likely to be considered 'innovative.'[http://74.125.155.132/scholar?q=cache:Ro0Ij3Y41wMJ:scholar.google.com/&hl=en&as_sdt=40000000&sciodt=40000000] Others focus more broadly on how to use the crowd to maintain or bolster business or brand. In ''Groundswell'' (2008),[http://books.google.com/books?id=YOVuQFXNcP4C&printsec=frontcover&dq=groundswell&hl=en&ei=ZF_tTKjtLIL58Ab-tu26AQ&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCsQ6wEwAA#v=onepage&q&f=false] Charlene Li and Josh Bernoff focus on how to most effectively use crowdsourcing to advantage businesses. The authors highlight how user bases of products can undermine a product or brand.[http://books.google.com/books?id=YOVuQFXNcP4C&printsec=frontcover&dq=groundswell&hl=en&src=bmrr&ei=tUPtTPOeAYP-8AbcxemdAQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCgQ6AEwAA#v=onepage&q&f=false] As a result, the authors propose that businesses use the "groundswell" to their advantage, fostering communities that ~~can~~ provide valuable feedback and economic payoffs. Marion K. Poetz and Martin Schreier have also taken a business perspective on crowdsourcing,[http://ssrn.com/abstract=1566903] arguing that the crowd is capable of producing valuable (but not always viable) business ideas at a low cost. Other researchers have found that young entrepreneurs who were attempting to start businesses frequently belonged to these kinds of communities.[http://ssrn.com/abstract=1022358] For a related discussion on user innovation and user communities, see Eric Von Hippel's books[http://web.mit.edu/evhippel/www/books.htm] and William Fisher's article in the Minnesota Law Review.[http://www.minnesotalawreview.org/content/implications-law-user-innovation]		While some focused on the potential consumer revolution or the composition of the crowd, others examined the business-related aspects of crowdsourcing. Identifying attributes of successful crowd innovators also has a business dimension. One researcher suggests that having experience spanning across a variety of communities or disciplines makes one likely to be considered 'innovative.'[http://74.125.155.132/scholar?q=cache:Ro0Ij3Y41wMJ:scholar.google.com/&hl=en&as_sdt=40000000&sciodt=40000000] Others focus more broadly on how to use the crowd to maintain or bolster business or brand. In ''Groundswell'' (2008),[http://books.google.com/books?id=YOVuQFXNcP4C&printsec=frontcover&dq=groundswell&hl=en&ei=ZF_tTKjtLIL58Ab-tu26AQ&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CCsQ6wEwAA#v=onepage&q&f=false] Charlene Li and Josh Bernoff focus on how to most effectively use crowdsourcing to advantage businesses. The authors highlight how user bases of products can undermine a product or brand.[http://books.google.com/books?id=YOVuQFXNcP4C&printsec=frontcover&dq=groundswell&hl=en&src=bmrr&ei=tUPtTPOeAYP-8AbcxemdAQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCgQ6AEwAA#v=onepage&q&f=false] As a result, the authors propose that businesses use the "groundswell" to their advantage, fostering communities that [http://167.114.87.240/ Judi Online] provide valuable feedback and economic payoffs. Marion K. Poetz and Martin Schreier have also taken a business perspective on crowdsourcing,[http://ssrn.com/abstract=1566903] arguing that the crowd is capable of producing valuable (but not always viable) business ideas at a low cost. Other researchers have found that young entrepreneurs who were attempting to start businesses frequently belonged to these kinds of communities.[http://ssrn.com/abstract=1022358] For a related discussion on user innovation and user communities, see Eric Von Hippel's books[http://web.mit.edu/evhippel/www/books.htm] and William Fisher's article in the Minnesota Law Review.[http://www.minnesotalawreview.org/content/implications-law-user-innovation]

	Other authors have pointed out some of the problems with crowdsourcing. Dr. Mathieu O'Neil has argued that, despite its benefits, crowdsourcing can have inconsistent quality, can lack the diversity needed to draw on the "wisdom of the crowd", and can contain many irresponsible actors.[http://www.paris-sorbonne.fr/fr/IMG/pdf/oneil-2.pdf] Miriam Cherry has argued that some crowdwork can be exploitative, sometimes forcing people to work for absurdly low wages.[http://ssrn.com/abstract=1499823] She argues that we need a legal framework for addressing low wages, proposing we apply the Fair Labor Standards Act (FLSA) to crowdsourced work like that found on Mechanical Turk. In a forthcoming article, she takes a more systematic (but still legal) approach to suggesting solutions for the problems faced by different kinds of virtual work.[http://ssrn.com/abstract=1649055] Cherry seems to be the only law professor to have written on addressing crowdsourcing from a doctrinal perspective.		Other authors have pointed out some of the problems with crowdsourcing. Dr. Mathieu O'Neil has argued that, despite its benefits, crowdsourcing can have inconsistent quality, can lack the diversity needed to draw on the "wisdom of the crowd", and can contain many irresponsible actors.[http://www.paris-sorbonne.fr/fr/IMG/pdf/oneil-2.pdf] Miriam Cherry has argued that some crowdwork can be exploitative, sometimes forcing people to work for absurdly low wages.[http://ssrn.com/abstract=1499823] She argues that we need a legal framework for addressing low wages, proposing we apply the Fair Labor Standards Act (FLSA) to crowdsourced work like that found on Mechanical Turk. In a forthcoming article, she takes a more systematic (but still legal) approach to suggesting solutions for the problems faced by different kinds of virtual work.[http://ssrn.com/abstract=1649055] Cherry seems to be the only law professor to have written on addressing crowdsourcing from a doctrinal perspective.
Line 31:		Line 31:
	6. Privacy Protection: Workers are concerned with employers sharing their (potentially sensitive) information, so platforms should protect information and not release it.		6. Privacy Protection: Workers are concerned with employers sharing their (potentially sensitive) information, so platforms should protect information and not release it.

	The Best Practices document provides a good starting point because it identifies several major issues common to all crowdsourcing. It does not, however, capture all potential problems. Additionally, it tends to focus concerns only on the workers, but platforms and companies also face similar problems. Moreover, because the document is meant as a general framework, it is hard to get a sense of whether it could be effectively implemented across the board. There is room, then, to explore problems that are both broad enough to have implications for a variety of actors, but specific enough to merit a context-specific solution.		The Best Practices document provides a good starting point because it identifies several major issues common to all crowdsourcing. It does not, however, capture all potential problems. Additionally, it tends to focus concerns only on the workers, but platforms and companies also face similar problems. Moreover, because the document is meant as a general framework, it is hard to get a sense of whether [http://142.44.144.187/ Situs Judi] could be effectively implemented across the board. There is room, then, to explore problems that are both broad enough to have implications for a variety of actors, but specific enough to merit a context-specific solution.

	== Our Addition: Identifying Areas, Exploring the Problems ==		== Our Addition: Identifying Areas, Exploring the Problems ==

	Given the body of literature and the Best Practices document, we found the idea of addressing systemic problems both attractive and difficult. Instead of replicating the Best Practices, or simply writing an overview of crowdsourcing, we decided to take a different angle. Unlike the Best Practices document, which classified problems generally and then worked downward to devise specific solutions by applying them to different types of crowdwork, we worked from the bottom up. We identified three types of crowdwork that suggested a variety of important, but (context-)specific problems. At the beginning stages, we had only our intuition to guide our "sense" of the problems. As we delved further into them, however, they crystallized. From ~~our~~ discussions we identified three types of crowdwork in which specific problems arise, some of which are systemic problems with crowdsourcing that the Best Practices does not address. Nevertheless, we wanted to draw on the Best Practices document to determine whether some of its strategies seemed workable or needed to be expanded, refined, or discarded. To accomplish this goal, we attempted to integrate the Best Practices approaches into our framing of both the problems and the solutions we discussed.		Given the body of literature and the Best Practices document, we found the idea of addressing systemic problems both attractive and difficult. Instead of replicating the Best Practices, or simply writing an overview of crowdsourcing, we decided to take a different angle. Unlike the Best Practices document, which classified problems generally and then worked downward to devise specific solutions by applying them to different types of crowdwork, we worked from the bottom up. We identified three types of crowdwork that suggested a variety of important, but (context-)specific problems. At the beginning stages, we had only our intuition to guide our "sense" of the problems. As we delved further into them, however, they crystallized. From [http://144.217.203.27/ Finalbet88] discussions we identified three types of crowdwork in which specific problems arise, some of which are systemic problems with crowdsourcing that the Best Practices does not address. Nevertheless, we wanted to draw on the Best Practices document to determine whether some of its strategies seemed workable or needed to be expanded, refined, or discarded. To accomplish this goal, we attempted to integrate the Best Practices approaches into our framing of both the problems and the solutions we discussed.

	=== An Introduction to Our Approach ===		=== An Introduction to Our Approach ===
Line 47:		Line 47:
	3. "Game" tasks: ''Gwap''[http://www.gwap.com/gwap/].		3. "Game" tasks: ''Gwap''[http://www.gwap.com/gwap/].

	For each of these tasks, we attempted to identify salient "problems": issues that cause concern for workers, employers, platforms, businesses, or society generally. In identifying problems, we had two goals. The first was to provide a set of new issues for others to build upon in future work. The second was to explore a small number of issues and propose our own context-specific solutions. In this sense, it was an exercise in both applying the Best Practices and inventing new solutions that either context or framing prevented the Best Practices from solving. In what follows, we explain each topic, the problems it presents, and specific solutions to selected problems. Although we think the solutions we propose have some teeth, they are not meant to be final. Indeed, our goal in presenting these solutions and problems is to provide a base from which ~~others~~ can build.		For each of these tasks, we attempted to identify salient "problems": issues that cause concern for workers, employers, platforms, businesses, or society generally. In identifying problems, we had two goals. The first was to provide a set of new issues for others to build upon in future work. The second was to explore a small number of issues and propose our own context-specific solutions. In this sense, it was an exercise in both applying the Best Practices and inventing new solutions that either context or framing prevented the Best Practices from solving. In what follows, we explain each topic, the problems it presents, and specific solutions to selected problems. Although we think the solutions we propose have some teeth, they are not meant to be final. Indeed, our goal in presenting these solutions and problems is to provide a base from which [http://139.99.117.208/ IDN POKER] can build.

	=== The 3 Crowdsourcing Environments and Problems ===		=== The 3 Crowdsourcing Environments and Problems ===