MediaCloud

From Berkman Klein Google Summer of Code Wiki
Revision as of 12:13, 16 March 2017 by BerkmanSysop (talk | contribs)
Jump to navigation Jump to search

Media Cloud is an open source platform for studying media ecosystems.

By tracking hundreds of millions of stories published online or broadcast via television, our suite of tools allows researchers to track how stories and ideas spread through media, and how different corners of the media ecosystem report on stories.

Our platform is designed to aggregate, analyze, deliver and visualize information, answering complex quantitative and qualitative questions about the content of online media.

Aggregate. We have aggregated billions of online stories from an ever-growing set of 25,000 digital media sources. We ingest data via RSS feeds and a set of robots that spider the web to fetch information from a variety of sources in near real-time.

Analyze. To query our extensive library of data, we have developed a suite of analytical tools that allow you to explore relationships between professional and citizen media, and between online and offline sources.

Deliver and Visualize. Our suite of tools provides opportunities to present data in formats that you can visualize in your own interfaces. These include the use of graphs, geographic maps, word clouds, network visualizations.

Project Mentor: [lvaliukas@cyber.law.harvard.edu mailto:lvaliukas@cyber.law.harvard.edu]

Projects

Build a tool to do some cool visualizations

Problem Statement. Since 2008, we have collected more than a half billion news articles that we have post-processed and indexed. We know quite a lot about them -- which news articles were the most linked to from other similar articles, the most and least popular / influential articles (based on shares on Facebook, tweet count, or clicks on an article's Bit.ly shortened link), specific language and terms used to describe the subject matter in each of the articles, etc., and there's a lot of potential to learn much more. Can you use your design and coding skills to help us out in visualising some of this data, e.g. create a cool network map visualization tool?

Development Tasks
  • Build any visualization tool based on our extensive data and tool set:
    • Figure out what you'd like to visualise and how are you going to do it
    • Use Gephi, a tool of your choice, or create your very own tool to implement your visualisation


Create PostgreSQL-based job queue

Problem Description. In more than eight (or is it nine by now?) years since we've been running Media Cloud, we have tried multiple job queue tools (e.g. Gearman) that we could use for dividing and conquering our workload. Unfortunately, all the tools (including the current one -- go look into the codebase to figure out which one it is now) have left us deeply unhappy because of one reason or another. If there's one tool which hasn’t let us down, it’s PostgreSQL. So, we'd like to also try running our job queue on Postgres. Can you implement it for us?

Development Tasks
  • Write a spec, complete with code samples, on how to implement the following job queue:
    • Preferably programming language-agnostic, i.e. should run as a bunch of PL/pgSQL functions.
      • Maybe that's a bad idea, I don't know, you tell us.
  • Features:
    • Add jobs with names and JSON arguments
    • Cancel jobs by their ID
    • Track job's progress (and log?) by their ID
    • Get job ID using its JSON parameters
    • Merge jobs with identical JSON arguments into a single job
    • See job stats per task, i.e. how many jobs are queued for every task
    • Retry failed jobs
    • Report job failure, complete with error messages
    • Proper locking (for inspiration, see https://github.com/chanks/que)
    • Doesn't catch fire with tens of millions of queued jobs
  • (Bonus points) Actually implement the queue! If you don't get to doing this over the summer, it's fine, we would be happy with a proven spec.


Implement topic creation using machine learning

Problem Statement. Our main tool for analyzing the vast amount of news articles that we have collected is what we call a "topic mapper". Topic mapper operates with "topics" which are subjects discussed by the media that we would like to research (e.g. the most recent US election, or Zika virus, or the right to be forgotten). Each topic has a rather simple boolean query (e.g. "right to be forgotten" OR "right to delist" OR "right to erasure" OR "right to oblivion") that the topic mapper uses to find news articles related to the media subject. Can we avoid writing those queries by making use of machine learning and just letting the server figure out which news articles are related to the topic by itself?

Development Tasks:
  • Develop a proof of concept (un)supervised ML tool for finding news articles on a chosen subject ("topic").

Containerize Media Cloud using Docker / Kubernetes / Mesos

Problem Description. Right now, we deploy our code using a rather archaic "SSH into server(s), type in some funky commands, press Return, pray" method. We've heard that oh-so-fancy tools like Docker allow one to containerize applications, thus introducing some nice opportunities to speed up deployments, isolate apps and resources, etc. We need to be shown how this might improve our life, and if we buy it, we'd like you to do it over the summer for us.

Development Tasks:
  • Convince us that we need Docker (or Kubernetes, or Mesos, or whatever).
  • Come up with a plan on how to migrate live production environment to the chosen containerization tool:
    • Which Media Cloud processes (crawler, extractor, database, etc.) should run in separate containers
    • Prerequisites
    • Timeline
    • Something else?
  • Prepare for running Media Cloud in a container:
    • Split Media Cloud codebase into separately deployable containers
    • Develop configuration, tooling, monitoring and documentation needed for deploying and maintaining containers
    • Something else?

Investigate moving full-text searches from Solr to PostgreSQL

Problem Description. We currently export all news articles from PostgreSQL to Solr to be able to do full-text searches among them. We have tried doing FTS with PostgreSQL itself years ago, but found out that it's too slow. Maybe with PostgreSQL 9.6 (or even 10) it would be a better idea?

Development Tasks:
  • Figure out the feasibility of moving full-text story searches from Solr to PostgreSQL:
    • Should support multiple languages (stopword removal, stemming, …)
    • Should support basic boolean operators
    • Should work with 500m+ news articles

Do your own freehand project

Problem Statement. If you had more than half a billion (!) news articles from all around the world stored in a single place, extracted from HTML into text, split into sentences, words, and made searchable, what would you do? Propose us something we didn't think of, and we will surely consider it!

Development Tasks.

Left as an exercise to the student.

Requirements

Retrieved from "http://cyber.harvard.edu/gsoc/?title=MediaCloud&oldid=532"