CrowdConf Brainstorm page

From Identifying Difficult Problems in Cyberlaw
Revision as of 21:43, 3 October 2010 by Jskoff (talk | contribs)
Jump to navigation Jump to search

Use this page to discuss the best practices reading we did not have time for in class, and brainstorm questions and topics that we might present as a class at the CrowdConf Future of Work Conference next week.

Crowdsourcing (Human Computing) is one of the most promising technologies, which has already been successfully in many different areas (Examples: X-Prize, 99designs, Amazon Mechanical Turk) and we believe has a huge potential for the future. It has the potential to significantly shape and change the way the labor market works. That said, it also creates challenges which need to be addressed. We would love to hear your thoughts on how technology could be leveraged to solve some of these challenges:

  • (1) Preserving Confidentiality in Complex Tasks. As the best practices document notes, some tasks require worker exposure to proprietary information. The Best Practices mention contracts as a way of dealing with this issues. Do we think that contractual relationships can assuage companies' fears of workers disclosing propriety information? Does the sheer volume (and potential geographical spread) of workers on a given task make enforcing such an agreement impossible?
    • Is there a way the technology can account for this problem?
    • Could the problem be solved potentially by drafting specific tasks to specific information, the disclosure of which would make the individual who divulged the info identifiable?
    • What are the costs of drafting such complex contracts?
  • (2) Feedback Mechanisms and Dispute Resolution. When there is little or no direct contact between employers and employees and when tasks are completed on a one-off basis, it can be tough to encourage fair feedback or to verify a potential worker's competence in advance. Workers themselves face portability problems; a good rating on Mechanical Turk doesn't necessarily carry over to other crowdsourcing sites or offline careers.
    • Could the technology facilitate a cyber dispute-resolution forum? (What if the dispute-resolution process was, in turn, crowd-sourced?!).
    • Could the platform have a rating system that suggested a fair rate based on the type of tasks requested? There could be a "survey" that each employer fills out before submitting the task, which would calculate a suggested rate. Perhaps it could be based off of past rates, as tracked by the platform operator? (Does Amazon's "recommended" technology do this in a different form already?)
    • Is there any way to use technology to prevent abuse of feedback systems, or at least encourage people to use the feedback system in good faith?
    • Have platforms set up features to facilitate the creation of online worker unions? (See SECTION BELOW for more questions on Online Worker Unions)
  • (3) Disclosure. The anonymity of cyber-space and the possibility to divide a large project into a a large number of small tasks so that the ultimate product is unidentifiable raises a number of ethical concerns. Have companies, clients and platforms alike, explored setting up or mandating an ethical commission investigating these concerns? What about a voluntary code of conduct created and agreed on by the industry as a quality management system to prevent black sheep from ruining the reputation of the entire industry in case of misconduct and as a preemptive action towards governmental regulation? --> How do you prevent a private-run "Manhattan Project" implemented through crowdsourcing and sold to the highest bidder?
    • (Zara) If project identification is important to the worker, would it help if cloudwork websites rated projects by their degree of macro-transparency and distinguished accordingly in their user interface? The ethical disclosure problem has at least three dimensions: 1) a worker's subjective discomfort with not knowing the purpose of their HIT; 2) the decision a worker might have made had they been able to make an informed choice; and 3) objective, ethical standards that should prevail in the industry. Although users can already select a HIT based on their assessment of the project, including the degree of information disclosed, organizing HITs by level of transparency could facilitate user choice that selects specifically for this criterion. By creating two portals, or databases of HITS, classified according to levels of transparency, disclosure would be highlighted as a desirable HIT characteristic as to which workers can and should exercise choice. If the database of high-disclosure HITs consistently generated higher quality workers and a reputation for legitimacy, normative pressures could encourage companies to offer more information about a task’s role in a larger project and the corresponding ability to compete in the high-disclosure market.


Follow-Up Questions / Further discussion points

  • Online Worker Unions. Crowdsourcing's success is dependent on finding ways to engage its labor pool, whether it be through offering money or gamesque points. However, as mentioned in class and in the best practices document, there are many ways for these laborers to become dissatisfied with their work, whether it be through a lack of transparency, stress, low wages, etc. Is there a potential for a crowdsourcing labor movement in response to these dissatisfactions? As an inherently digital workforce, these individuals' attempts to share discontents and act upon them are facilitated by their familiarity with and access to online communities. However, how far will this unity go? Do you feel that workers will only offer critiques of certain employers to others or could there be the formation of unions and similar entities in the crowdsourcing world?
  • Overlap of legal frameworks. Some countries have a state pension fund that is financed by a tax deducted from a worker's salary. How are these legal requirements adhered to in the realm of crowd-sourcing? How is the location/jurisdiction of the worker determined? If the company's location is chosen, how measures are taken for the worker to have access to that respective legal system?
  • Compensation. Crowdsourcing appears to rely on monetary compensation, a gamelike points system, or personal gratification to motivate people to participate in these tasks. Which of these compensation forms is the most effective in ensuring a large labor pool and the best results for employers? Which (if any) of these forms will be the most prominent system of compensation in the future and which do you think would be the most ideal compensation structure for crowdsourcing in the future?
    • Virtual Currency Concerns
      • A growing trend is for companies to pay "cloudworkers" in virtual goods, rather than real money, and this raises some interesting concerns. For example, CrowdFlower (in parternship with game maker Zynga) pays around 100,000 workers in virtual goods rather than real money -- which the gamers can then use to buy things such as virtual seeds for Farmville. CrowdFlower CEO Lukas Biewald told BusinessWeek [1] that demand for virtual rather than real wages skews young: "There will be a whole new generation of kids growing up who won't really see the difference." Yet what is the difference?
      • Does payment in virtual goods attract more children to cloudwork? How does payment in virtual goods impact laws governing "real" work (for real currency)? Do game makers who offer cloudwork risk harming players? For example, if Zynga offered a weapon in Mafia Wars for performing 10 hours of cloudwork, yet made this same weapon difficult to otherwise obtain, one can imagine that players might feel pressured (or exploited) into performing cloudwork. Have there been any such instances?
  • Employment relationship". Are all cloudworkers to be considered independent contractors? Does this qualification pass the test of current labour laws and tax laws?
    • (Zara) The IRS definition of an employee versus an independent contractor, which tries to codify the common law, seems inapplicable to the cloudworker-employer relationship. See Independent Contractor (Self-Employed) or Employee? The page on independent contractors states: "The general rule is that an individual is an independent contractor if you, the person for whom the services are performed, have the right to control or direct only the result of the work and not the means and methods of accomplishing the result." [2] As Davis observes in his discussion of scientific research, cloudsourcing problems tend to be idea-based. Applying the common law definition to idea-oriented micro-tasks, which don't require specialized instrumentalities or complex services, is difficult. It could be argued that requestors control the means and methods of accomplishing the results by virtue of the nature of the task; the task can only be accomplished in readily identifiable ways. But if the "means and method" of accomplishing a result is the process of thinking, then an employer will have a hard time establishing this level of control!
    • In searching the internet I found this interesting blog post (http://behind-the-enemy-lines.blogspot.com/2009/07/is-amazon-mechanical-turk-black-market.html). I do not know if the information provided is accurate, but it raises some interesting questions: how do crowdsourcing platforms respond to the accusation of fostering a black market?
    • Is it appropriate for crowdsourcing platforms to construe a relationship between the worker and the requester? Or should any responsibility be allocated to such platforms?
    • Should the role of the platforms be more active in ensuring fairness of the working conditions and salary?
    • Crowdsourcing platforms have sometimes been depicted as virtual sweatshops. Do platforms feel any pressure from this criticism? Will increasing awareness of this perception in the general public prompt crowdsourcing platforms to adopt more stringent policies for the monitoring of working conditions?
    • Should minimum wages be granted to full-time cloudworkers? Should the maximum working hours be monitored and controlled by the platform? It has been argued that the Fair Labor Standards Act should apply to crowdsourcing platforms (Working For (Virtually) Minimum Wage: Applying the Fair Labor Standards Act In Cyberspace; 60 Ala. L. Rev. 1077). What is the platforms' response to this?
    • How crowdsourcing platforms address the issue of labour conditions in developing countries? Is the age of the cloudworker actually checked in order to avoid children labour exploitation? Should it be? And how?


  • Mobile Online Devices. Mobile and closed platforms with constant connection to the Internet have been supplanting sedentary workplaces in popularity.. How has crowdsourcing taken advantage of this change or has it struggled to do so? What advantages/challenges do these mobile devices offer workers, employees, and crowdsourcing agencies?
  • Recommendation System / Performance Tracking. It seems like sharing information about workers as suggested in the Best Practices document is a bit invasive for my taste, and perhaps something would have to be written into workers' contracts to explicitly allow this type of information sharing? (I'm not exactly what sure this law would implicate, but I seem to remember that when a potential employer calls a jobseeker's former employers, the employers can only share information that the person worked there, but can't reveal performance evaluations, etc.) Perhaps it's just me, but I'd feel more comfortable if companies enabled performance-tracking software, but didn't go so far as to share it with all other similar companies. (Question from Jenny)
  • Encryption? In some cases, the employer might want to protect the large picture of the project from workers while still makes each piece of task workable. For example, a writer might want his article to be shortened or proofread, but not necessarily want the workers to get the idea of the article before it gets published. How to make sure that they would not be able to do so (by e.g. divide the project into extremely small pieces, or, cut off possible communications among co-workers)?
    • Secrecy in projects, however, also implicates the downsides with making discrete jobs that cover up the underlining project purpose. The Best Practices discussed some of these with micro-tasks (e.g. moral qualms with the project and lower quality of performance). Also, the employees will be isolated by such secret tasks since they will both lack a physical place to work with other employees and be cut off from discussing it in crowdsourcing chat rooms. One thing then to think about then is how do you balance those two objectives to get the best overall outcome for a specific project (thinking about desired secrecy and worker performance/satisfaction almost as two axes on a graph)? Again, as mentioned above, would developing contractual relationships to deal with this suffice?
  • Quality control. As the demonstration showed last week, the author still needs to use his own judgment to eliminate many inappropriate revisions suggested by the crowd. Is it possible to develop certain methods to have different pools of people to whom different projects can be assigned? Without collecting too much private information, we can use demographic standards to differentiate the crowd (age, sex, education, etc). Then the quality of collaborative product can be raised and author’s final intervention can also be reduced.
    • In thinking about pulling from different pools of crowdsourcing employees, I’d be curious to ask about employee information verification. Basically, when recruiting crowdsourcing employees, especially when seeking out a certain educational background, how difficult is it to verify that they in fact have that background? Is misrepresentation of that kind of information more prevalent in the crowdsourcing context since it is largely faceless without personal interaction to help catch people in a lie? If so, would it be (or has it been shown to be) helped by the improved work environment ideas discussed in the Best Practices, such as chat rooms for employees that simulate a virtual water cooler?

(Sorry, forgot to log in, this is Erin) OK so to keep in mind what our goal is — we're supposed to put together some sort of agenda to talk about with the people who think about this day-in and day-out, and we have about half an hour? So it seems like we should try to focus in on a particularly compelling angle. The list above is good but can we prioritize? I really liked the point made in class last week that pointed out that if we can identify some sort of way for interesting technology to "fix" the "problems" that we see arising out of crowdsourcing, we'll have a much more receptive audience. Nothing jumps out at me from any particular section of the Best Practices document, but if we combine some of it maybe we can come up with something interesting?

  • Maybe combining some of the aspects of portability and reviews with the identity movement more generally would be interesting?
  • Frankly a lot of the best practices aren't super interesting in terms of the required technology — is there some other way to get them excited about a particular angle on something?
  • Is there a way to frame a problem that we're particularly concerned about that will speak to them? Don't mean to make this an "us-against-them" thing — but the way that technologists think about technology is a little different from the way that lawyers do, so we want to be able to frame the issue in a way that will resonate with the audience...
    • What about praising the technology (maybe have a few specific examples), but then asking them if they've met any resistance from, or thought about, any of the potential actors who may block or alter the technology's use? Have you (the technologist) talked with local/state/federal government? Have you discussed potential roadblocks with companies that will implement the technology?

Crowdsourcing Science

Jenny here: I've been reading some blog posts on crowdsourcing, and one comment from a scientist (found in the comments section here: http://money.usnews.com/money/blogs/outside-voices-small-business/2009/01/27/using-social-media-and-crowd-sourcing-for-quick-and-simple-market-research ) got me thinking that scientific research and development could suffer if companies move from hiring a dedicated team of scientists to farming all of their scientific problems out to a crowdsourcing lottery payment system (ie, first one to do this gets all of this money; the others get nothing). Honestly, worst case scenario, we'd have even fewer people going into sophisticated scientific fields than we do now, because there wouldn't be any guarantee of a stable living, and I wonder if this could really hinder the development of solutions to scientific problems or if it would limit the scientific fields to scientists who are business-savvy enough to be connected to venture capitalists, etc. Either way, the outcome could be scary.

I'd be interested to hear from crowdsourcing experts about how they think crowdsourcing scientific problems affects the quality of scientific research, and if there could be any safeguards implemented to prevent the aforementioned problems from occurring (ie -- could the crowdsourcing community fund a dedicated pool of scientists, with extra prizes going to those who successfully complete R&D tasks, or would this go against the very core of the crowdsourcing movement?)

  • (Heather): I'm not sure this is a novel problem in the crowdsourcing context. I've never worked in a lab or been involved in research, but this concern about stable living and a consistent funding source seems to be pretty common (at least in the academic context, where it seems like a lot of time and energy is spent chasing down grants and competing for funding). Obviously the decentralization of oDesk and similar websites exacerbates the problem by breaking down competition into individual tasks and individual workers instead of across entire projects or teams, but it doesn't seem to me to create an entirely new economic incentive structure. Then again, I'm not an expert on this and we'll be talking to people who are.
  • (Davis): I question whether crowdsourcing has the potential to displace much scientific research. Most commercially viable research projects (such as pharmaceuticals) require significant capital investments in sophisticated experimental equipment, or access to tightly regulated materials (such as dangerous chemicals or radioactive sources). There is simply no way to crowdsource around the need for a spectrometer or a chromatograph. The types of scientific problems that are readily solved through crowdsourcing will tend to be idea-based (rather than experiment-based), and correct solutions must be easily verifiable. These criteria alone suffice to tightly restrict the class of problems that are amenable to solution through, say, Innocentive. Moreover, companies will need to employ scientific experts simply to know what questions to ask (and how to divide larger problems into smaller ones), and so a significant amount of centralization will still be necessary even with distributable projects.
    • And of course much scientific research is basic research, and therefore not (immediately) commercially viable. Thus we're unlikely to see a large category of research go the way of Innocentive. Something like the Large Hadron Collider is the very antithesis of crowdsourcing; such large collaborative projects seem to be the direction physics research will be headed for some time to come. The same will probably become true for other scientific fields as they mature.
    • (As a side note, I also feel the need to add that the concern we often hear about having an insufficient number of people pursuing the hard sciences is overblown. In most fields, there simply aren't enough jobs out there to absorb the quantity of science PhDs we produce.)
  • (Rachel): As an offshoot of what Davis said, I'm curious as to whether we can discern general principles of when crowdsourcing is or is not viable -- not so much in terms of Rene's question about public acceptance of crowdsourcing, but rather in terms of when it can actually be done or not, as in Davis' Large Hadron Collider example. Or what about litigation? Doc review is being outsourced more and more to contract lawyers working as independent contractors, both within the U.S. and abroad, so that seems like fair game for crowdsourcing (assuming we can get 'specialized' crowds), but it does not seem like the same could be said for actual trial or appellate practice in court. Is it merely a question of skill and experience? Given projects like Innocentive, where scientific issues requiring a lot of skill and experience are crowdsourced, that does not seem to be the case, and yet what is it about things like Large Hadron Colliders and trials that appear to be resistant to crowdsourcing?

Case Study: oDesk (Rene)

  • I have used oDesk a lot of times over the summer to outsource smaller programming projects for my startup to developers, mainly in India and Southeast Asia. For those who haven't used oDesk, you post a job with a budget, oDesk workers apply for the job, you can interview them and then hire one; a small portion of the overall payment might be upfront, the rest is paid at completion of the project at discretion of the employer. oDesk has standard terms (NDA, etc.) to facilitate the transactions, but I have asked the developers I hired to sign additional documentation. The biggest issue is quality control; despite the fact that there is a rating system, it is quite difficult to evaluate whether someone is able to get a certain job done or not. I really like Jenny's question around recommendation systems / quality control as and extension of point (2) above and would like to hear what technologist have in mind to address this important challenge.

I would like to hear a discussion about the general public's acceptance of crowdsourcing. As mentioned in class, our knowledge and opinions of crowdsourcing is very much a minority viewpoint. Although to us it presents a really novel and theoretically interesting development, I imagine different entities (investors, crowdsourcing employees, workers outside of the field) would view this new practice through the lens of their own interests. I would like to hear these crowdsourcing leaders discuss their interactions with these groups, either through an open question or a directed one.

  • Can a parallel be made to outsourcing? When jobs are outsourced, domestic workers feel outrage and a sense that they are being cheated or dealt with unfairly. One can imagine a similar response on a micro-level to crowdsourcing, where formerly dedicated employees are let go in favor of crowdsourcing, but employers are happy because it is more economically efficient.
  • If people have made parallels to outsourcing, is that even a fair parallel to make? One can certainly make an argument that the small tasks being accomplished by 100s of people are not necessarily displacing jobs on a 1 to 1 level. Also, these jobs are not disappearing from the United States entirely. If your job has been replaced by crowdsourcing, you could just become one of the crowd at that point and do tasks for a wider array of companies. Are claims of outsourcing just knee jerk reactions to something that actually has the potential to offer the same amount or more opportunities for workers?

Lessig Framework

What if we approach the best practices document with a view to Lessig's four modes of regulation, and frame our discussion of crowdsourcing in terms of which combination of modes could best achieve the desired outcomes? For example, assume a crowdsourcing application that has an architecture in place forcing disclosure pursuant to the best practices model. With such a system in place, norms may then provide the best solution to the fairness problem: workers would share information about employers who are known to violate users' sense of fairness in worker forums, and discourage others from doing the work. Or workers could "strike" by making a concerted effort to accept all that employer's tasks and intentionally perform poorly, thereby obstructing completion of the disfavored company's assignments (sort of like 4Chan meets Mechanical Turk).

On a related question, could a crowdsourcing approach solve any of the crowdsourcing best practices problems? For example, is there a way to implement a feedback and monitoring system whereby the quality of a submission is judged by crowd workers?

Mobile Online Devices

This probably won't appeal to the tech crowd as much, but I'm interested in how we will deal with the mobile devices problem mentioned above. If you're working from your phone, which jurisdiction's labor laws apply? Is it the state where your phone was issued, the state where you live, or the state where you're using the device?

  • How can companies protect themselves when they want crowdsourcing work done, but want to avoid certain states?
    • i.e. if I want to avoid CA labor laws, I can block CA IP addresses, but that wouldn't necessarily prevent me from taking my TX iPhone and working for a crowdsourcing company when I'm in CA.
  • What about international workers?
  • Should we use architecture by forcing workers to log in through a mobile application that requires assent to certain terms (i.e. I'm not working in CA)?
Retrieved from "http://cyber.harvard.edu/difficultproblems/?title=CrowdConf_Brainstorm_page&oldid=177"