MediaCloud

Media Cloud is an open source platform for studying media ecosystems. Media Cloud is joint project between The Berkman Klein Center for Internet and Society at Harvard University and The Center for Civic Media at MIT's Media Lab.

By tracking hundreds of millions of stories published online or broadcast via television, our suite of tools allows researchers to track how stories and ideas spread through media, and how different corners of the media ecosystem report on stories.

Our platform is designed to aggregate, analyze, deliver and visualize information, answering complex quantitative and qualitative questions about the content of online media.

• Aggregate: We have aggregated billions of online stories from an ever-growing set of 1,200,000+ digital media sources. We ingest data via RSS feeds and a set of robots that spider the web to fetch information from a variety of sources in near real-time.

• Analyze: To query our extensive library of data, we have developed a suite of analytical tools that allow you to explore relationships between professional and citizen media, and between online and offline sources.

• Deliver and Visualize: Our suite of tools provides opportunities to present data in formats that you can visualize in your own interfaces. These include the use of graphs, geographic maps, word clouds, network visualizations.

Project URL: https://mediacloud.org/

Project on GitHub: https://github.com/berkmancenter/mediacloud

Project Mentors: Linas Valiukas, Hal Roberts

Project Ideas

Create a self-contained, browser-based page HTML -> article HTML extractor

Problem Statement: For every fetched news article, we have to figure out which part of the page HTML contains the article body itself. We currently use readability-lxml (https://github.com/buriy/python-readability) for that task. However, readability-lxml is aging fast and is not necessarily still the best library around to extract body of the article from the HTML page. Also, more and more articles get loaded using JavaScript due to an ongoing "frontend everywhere!" frenzy, and our Python extractor doesn’t execute or support JavaScript. Lastly, various CDNs, e.g. Cloudflare, are blocking our crawler just because our user agent doesn't have JavaScript enabled.

I think inevitably we'll have to switch to running a headless browser, loading each and every downloaded story in it, and then applying a well-supported third-party library, e.g. Mozilla's Readability (https://github.com/mozilla/readability), to extract article title, author and body.

Development Tasks

• Set up a chromeless browser
• Set up Readability
• Develop a HTTP service that accepts a parameter URL (and/or HTML body), loads it in the browser, runs Readability's magic, and returns the extracted HTML back to the requester.
• Package everything in a nice Docker image

There are similar projects like this, e.g. https://github.com/schollz/readable, so you’ll need to do some research into whether such a thing exists already before submitting a proposal. Maybe an existing tool could be improved instead of redoing everything from scratch?

Write a spec to a new generation of our API

Problem Statement: Create a specification for a new version of our API (https://github.com/berkmancenter/mediacloud/tree/release/doc/api_2_0_spec). Our existing API (implemented in Perl) is inconsistent among its different major parts, and is goofily un-REST-ish in several places. We would like to reimplement it to Python and use a modern framework for API specification (OpenAPI), implementation, and testing.

Development Tasks

• With the help of the team, identify which API calls can be renamed to more sensible names, extended or deprecated
• With the help of the team, rewrite API call descriptions to make them more comprehensible
• Rewrite a API spec using a chosen tool (e.g. OpenAPI) using best RESTful practices
• Set up API demo (e.g. using Swagger UI)

Rewrite Ultimate Sitemap Parser to yield results instead of returning them

Problem Statement: Ultimate Sitemap Parser (https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser) is our Python module that we use to fetch trees of website sitemaps (https://www.sitemaps.org/). The current implementation of the sitemap parser fetches all of the sitemap links and returns it to a caller in a single easy-to-use object. However, it turns out that some websites have *massive* sitemap trees! In those cases, the sitemap parser uses up lots of RAM for its operation, and the client is forced to wait for a long time to get sitemap fetching + parsing results. For those reasons, we’d like the sitemap parser to “yield” links found in a sitemap instead of “returning” them while also maintaining a nice, comprehensible interface to the sitemap parser: https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser/issues/2

Development Tasks

• Rewrite sitemap parser to yield found sitemap links instead of returning them to conserve memory and make results usable faster

Make Ultimate Sitemap Parser use asyncio

Problem Statement: Ultimate Sitemap Parser (https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser) is our Python module that we use to fetch trees of website sitemaps (https://www.sitemaps.org/). In production, fetching and parsing XML sitemaps it’s mostly a CPU-intensive operation as the most time gets spent on gunzipping sitemaps, parsing XML and creating objects out of them, but my guess is that the sitemap parser could be made 10-20% faster by doing I/O (namely the fetching part) asynchronously.

Development Tasks

• Rewrite sitemap parser to fetch sitemaps asynchronously
• Find other ways where I/O could be made asynchronous

Detect sitemap if it’s not linked from robots.txt in Ultimate Sitemap Parser

Problem Statement: Ultimate Sitemap Parser (https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser) is our Python module that we use to fetch trees of website sitemaps (https://www.sitemaps.org/). Most of sitemaps are being linked to in website’s robots.txt, but some are not. We would like to try common paths of sitemaps (e.g. /sitemap.xml[.gz]) on every site nonetheless: https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser/issues/8

Development Tasks

• Update the module in such a way that it tries common sitemap locations independently from robots.txt

Add RSS / Atom sitemap support to our Ultimate Sitemap Parser

Problem Statement: Ultimate Sitemap Parser (https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser) is our Python module that we use to fetch trees of website sitemaps (https://www.sitemaps.org/). Most of those sitemaps are implemented in Sitemap XML protocol, but a small number of sitemaps are published in RSS / Atom, and we’d like to have support for those too: https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser/issues/3

Development Tasks

• Add RSS / Atom support to Ultimate Sitemap Parser

Build a tool to do some cool visualizations

Problem Statement Since 2008, we have collected more than a half billion news articles that we have post-processed and indexed. We know quite a lot about them -- which news articles were the most linked to from other similar articles, the most and least popular / influential articles (based on shares on Facebook, tweet count, or clicks on an article's Bit.ly shortened link), specific language and terms used to describe the subject matter in each of the articles, etc., and there's a lot of potential to learn much more. Can you use your design and coding skills to help us out in visualising some of this data, e.g. create a cool network map visualization tool?

Development Tasks

• Build any visualization tool based on our extensive data and tool set:
  • Figure out what you'd like to visualise and how are you going to do it
  • Use Gephi, a tool of your choice, or create your very own tool to implement your visualisation

Create PostgreSQL-based job queue

Problem Description. In more than eight (or is it nine by now?) years since we've been running Media Cloud, we have tried multiple job queue tools (e.g. Gearman) that we could use for dividing and conquering our workload. Unfortunately, all the tools (including the current one -- go look into the codebase to figure out which one it is now) have left us deeply unhappy because of one reason or another. If there's one tool which hasn’t let us down, it’s PostgreSQL. So, we'd like to also try running our job queue on Postgres. Can you implement it for us?

Development Tasks

• Write a spec, complete with code samples, on how to implement the following job queue:
  • Preferably programming language-agnostic, i.e. should run as a bunch of PL/pgSQL functions.
    • Maybe that's a bad idea, I don't know, you tell us.
• Features:
  • Add jobs with names and JSON arguments
  • Cancel jobs by their ID
  • Track job's progress (and log?) by their ID
  • Get job ID using its JSON parameters
  • Merge jobs with identical JSON arguments into a single job
  • See job stats per task, i.e. how many jobs are queued for every task
  • Retry failed jobs
  • Report job failure, complete with error messages
  • Proper locking (for inspiration, see https://github.com/chanks/que)
  • Doesn't catch fire with tens of millions of queued jobs
• (Bonus points) Actually implement the queue! If you don't get to doing this over the summer, it's fine, we would be happy with a proven spec.

Implement a method to detect subtopics of a topic

• Problem Statement.* As described elsewhere, a "topic" is subject discussed by the media that we are researching. Almost every big topic contains subtopics, e.g. the matters of immigration, racism, email server security and a plethora of other subjects were discussed during the last US election. We would like to investigate ways of how we could automatically detect those subtopics, possibly using the [Louvain method](https://en.wikipedia.org/wiki/Louvain_Modularity).

Development Tasks

Develop a proof of concept (un)supervised ML tool for detecting subtopics of a chosen subject ("topic").

Do your own freehand project

Problem Statement. If you had more than half a billion (!) news articles from all around the world stored in a single place, extracted from HTML into text, split into sentences, words, and made searchable, what would you do? Propose us something we didn't think of, and we will surely consider it!

Development Tasks.

Left as an exercise to the student.

Skill Requirements for Potential Candidates

• Working knowledge of Perl or Python
• Familiarity with relational databases, preferably PostgreSQL
• Some pedantism
• Willingness to propose, debate and object to ideas
• Keen to work with us on writing your GSoC project proposal, as opposed to just submitting a long shot without any feedback and hoping for the best
• Shown effort into learning what Media Cloud is all about; some ideas:
  • Make a pull request to our main code repository (https://github.com/berkmancenter/mediacloud),
  • Craft us an email with a smart question or two,
  • Try our our tools (see https://mediacloud.org/),
  • Run Media Cloud yourself and collect some news articles (see https://github.com/berkmancenter/mediacloud/blob/master/doc/vagrant.markdown),
  • Sign up and check out our API (see https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/api_2_0_spec.md, https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md, and the API client at https://pypi.python.org/pypi/mediacloud/).