MediaCloud

From Berkman Klein Google Summer of Code Wiki
Revision as of 16:25, 12 March 2018 by BerkmanSysop (talk | contribs)
Jump to navigation Jump to search

Media Cloud is an open source platform for studying media ecosystems.

By tracking hundreds of millions of stories published online or broadcast via television, our suite of tools allows researchers to track how stories and ideas spread through media, and how different corners of the media ecosystem report on stories.

Our platform is designed to aggregate, analyze, deliver and visualize information, answering complex quantitative and qualitative questions about the content of online media.

Aggregate. We have aggregated billions of online stories from an ever-growing set of 25,000 digital media sources. We ingest data via RSS feeds and a set of robots that spider the web to fetch information from a variety of sources in near real-time.

Analyze. To query our extensive library of data, we have developed a suite of analytical tools that allow you to explore relationships between professional and citizen media, and between online and offline sources.

Deliver and Visualize. Our suite of tools provides opportunities to present data in formats that you can visualize in your own interfaces. These include the use of graphs, geographic maps, word clouds, network visualizations.

Project Mentor: Linas Valiukas lvaliukas@cyber.law.harvard.edu

Projects

Build a tool to do some cool visualizations

Problem Statement. Since 2008, we have collected more than a half billion news articles that we have post-processed and indexed. We know quite a lot about them -- which news articles were the most linked to from other similar articles, the most and least popular / influential articles (based on shares on Facebook, tweet count, or clicks on an article's Bit.ly shortened link), specific language and terms used to describe the subject matter in each of the articles, etc., and there's a lot of potential to learn much more. Can you use your design and coding skills to help us out in visualising some of this data, e.g. create a cool network map visualization tool?

Development Tasks
  • Build any visualization tool based on our extensive data and tool set:
    • Figure out what you'd like to visualise and how are you going to do it
    • Use Gephi, a tool of your choice, or create your very own tool to implement your visualisation


Create PostgreSQL-based job queue

Problem Description. In more than eight (or is it nine by now?) years since we've been running Media Cloud, we have tried multiple job queue tools (e.g. Gearman) that we could use for dividing and conquering our workload. Unfortunately, all the tools (including the current one -- go look into the codebase to figure out which one it is now) have left us deeply unhappy because of one reason or another. If there's one tool which hasn’t let us down, it’s PostgreSQL. So, we'd like to also try running our job queue on Postgres. Can you implement it for us?

Development Tasks
  • Write a spec, complete with code samples, on how to implement the following job queue:
    • Preferably programming language-agnostic, i.e. should run as a bunch of PL/pgSQL functions.
      • Maybe that's a bad idea, I don't know, you tell us.
  • Features:
    • Add jobs with names and JSON arguments
    • Cancel jobs by their ID
    • Track job's progress (and log?) by their ID
    • Get job ID using its JSON parameters
    • Merge jobs with identical JSON arguments into a single job
    • See job stats per task, i.e. how many jobs are queued for every task
    • Retry failed jobs
    • Report job failure, complete with error messages
    • Proper locking (for inspiration, see https://github.com/chanks/que)
    • Doesn't catch fire with tens of millions of queued jobs
  • (Bonus points) Actually implement the queue! If you don't get to doing this over the summer, it's fine, we would be happy with a proven spec.


Containerize Media Cloud using Docker / Kubernetes / Mesos

Problem Description. Right now, we deploy our code using a rather archaic "SSH into server(s), type in some funky commands, press Return, pray" method. We've heard that oh-so-fancy tools like Docker allow one to containerize applications, thus introducing some nice opportunities to speed up deployments, isolate apps and resources, etc. We need to be shown how this might improve our life, and if we buy it, we'd like you to do it over the summer for us.

Development Tasks:
  • Convince us that we need Docker (or Kubernetes, or Mesos, or whatever).
  • Come up with a plan on how to migrate live production environment to the chosen containerization tool:
    • Which Media Cloud processes (crawler, extractor, database, etc.) should run in separate containers
    • Prerequisites
    • Timeline
    • Something else?
  • Prepare for running Media Cloud in a container:
    • Split Media Cloud codebase into separately deployable containers
    • Develop configuration, tooling, monitoring and documentation needed for deploying and maintaining containers
    • Something else?

Implement a method to detect subtopics of a topic

  • Problem Statement.* As described elsewhere, a "topic" is subject discussed by the media that we are researching. Almost every big topic contains subtopics, e.g. the matters of immigration, racism, email server security and a plethora of other subjects were discussed during the last US election. We would like to investigate ways of how we could automatically detect those subtopics, possibly using the [Louvain method](https://en.wikipedia.org/wiki/Louvain_Modularity).
Development Tasks

Develop a proof of concept (un)supervised ML tool for detecting subtopics of a chosen subject ("topic").

Do your own freehand project

Problem Statement. If you had more than half a billion (!) news articles from all around the world stored in a single place, extracted from HTML into text, split into sentences, words, and made searchable, what would you do? Propose us something we didn't think of, and we will surely consider it!

Development Tasks.

Left as an exercise to the student.

Requirements

Retrieved from "http://cyber.harvard.edu/gsoc/?title=MediaCloud&oldid=570"