MediaCloud - Revision history

BerkmanSysop: /* Build a tool to do some cool visualizations */

2019-04-02T18:37:34Z

Build a tool to do some cool visualizations

← Older revision		Revision as of 18:37, 2 April 2019
Line 86:		Line 86:

	==Build a tool to do some cool visualizations==		==Build a tool to do some cool visualizations==
	''Problem Statement'' Since 2008, we have collected more than a half billion news articles that we have post-processed and indexed. We know quite a lot about them -- which news articles were the most linked to from other similar articles, the most and least popular / influential articles (based on shares on Facebook, tweet count, or clicks on an article's Bit.ly shortened link), specific language and terms used to describe the subject matter in each of the articles, etc., and there's a lot of potential to learn much more. Can you use your design and coding skills to help us out in visualising some of this data, e.g. create a cool network map visualization tool?		'''Problem Statement''' Since 2008, we have collected more than a half billion news articles that we have post-processed and indexed. We know quite a lot about them -- which news articles were the most linked to from other similar articles, the most and least popular / influential articles (based on shares on Facebook, tweet count, or clicks on an article's Bit.ly shortened link), specific language and terms used to describe the subject matter in each of the articles, etc., and there's a lot of potential to learn much more. Can you use your design and coding skills to help us out in visualising some of this data, e.g. create a cool network map visualization tool?

	===Development Tasks===		===Development Tasks===
Line 92:		Line 92:
	**Figure out what you'd like to visualise and how are you going to do it		**Figure out what you'd like to visualise and how are you going to do it
	**Use Gephi, a tool of your choice, or create your very own tool to implement your visualisation		**Use Gephi, a tool of your choice, or create your very own tool to implement your visualisation



	==Create PostgreSQL-based job queue==		==Create PostgreSQL-based job queue==

BerkmanSysop at 18:36, 2 April 2019

2019-04-02T18:36:29Z

← Older revision		Revision as of 18:36, 2 April 2019
Line 5:		Line 5:
	Our platform is designed to aggregate, analyze, deliver and visualize information, answering complex quantitative and qualitative questions about the content of online media.		Our platform is designed to aggregate, analyze, deliver and visualize information, answering complex quantitative and qualitative questions about the content of online media.

	*Aggregate: We have aggregated billions of online stories from an ever-growing set of 1,200,000+ digital media sources. We ingest data via RSS feeds and a set of robots that spider the web to fetch information from a variety of sources in near real-time.		*'''Aggregate''': We have aggregated billions of online stories from an ever-growing set of 1,200,000+ digital media sources. We ingest data via RSS feeds and a set of robots that spider the web to fetch information from a variety of sources in near real-time.

	*Analyze: To query our extensive library of data, we have developed a suite of analytical tools that allow you to explore relationships between professional and citizen media, and between online and offline sources.		*'''Analyze''': To query our extensive library of data, we have developed a suite of analytical tools that allow you to explore relationships between professional and citizen media, and between online and offline sources.

	*Deliver and Visualize: Our suite of tools provides opportunities to present data in formats that you can visualize in your own interfaces. These include the use of graphs, geographic maps, word clouds, network visualizations.		*'''Deliver and Visualize''': Our suite of tools provides opportunities to present data in formats that you can visualize in your own interfaces. These include the use of graphs, geographic maps, word clouds, network visualizations.

	Project URL: https://mediacloud.org/		'''Project URL''': https://mediacloud.org/

	Project on GitHub: https://github.com/berkmancenter/mediacloud		'''Project on GitHub''': https://github.com/berkmancenter/mediacloud

	Project Mentors: [mailto:linas@media.mit.edu Linas Valiukas], [mailto:hroberts@cyber.law.harvard.edu Hal Roberts]		'''Project Mentors''': [mailto:linas@media.mit.edu Linas Valiukas], [mailto:hroberts@cyber.law.harvard.edu Hal Roberts]

	=Project Ideas=		=Project Ideas=

BerkmanSysop at 18:35, 2 April 2019

2019-04-02T18:35:51Z

← Older revision		Revision as of 18:35, 2 April 2019
Line 5:		Line 5:
	Our platform is designed to aggregate, analyze, deliver and visualize information, answering complex quantitative and qualitative questions about the content of online media.		Our platform is designed to aggregate, analyze, deliver and visualize information, answering complex quantitative and qualitative questions about the content of online media.

	Aggregate. We have aggregated billions of online stories from an ever-growing set of 1,200,000+ digital media sources. We ingest data via RSS feeds and a set of robots that spider the web to fetch information from a variety of sources in near real-time.		*Aggregate: We have aggregated billions of online stories from an ever-growing set of 1,200,000+ digital media sources. We ingest data via RSS feeds and a set of robots that spider the web to fetch information from a variety of sources in near real-time.

	Analyze. To query our extensive library of data, we have developed a suite of analytical tools that allow you to explore relationships between professional and citizen media, and between online and offline sources.		*Analyze: To query our extensive library of data, we have developed a suite of analytical tools that allow you to explore relationships between professional and citizen media, and between online and offline sources.

	Deliver and Visualize. Our suite of tools provides opportunities to present data in formats that you can visualize in your own interfaces. These include the use of graphs, geographic maps, word clouds, network visualizations.		*Deliver and Visualize: Our suite of tools provides opportunities to present data in formats that you can visualize in your own interfaces. These include the use of graphs, geographic maps, word clouds, network visualizations.

	Project URL: https://mediacloud.org/		Project URL: https://mediacloud.org/

BerkmanSysop at 18:35, 2 April 2019

2019-04-02T18:35:01Z

← Older revision		Revision as of 18:35, 2 April 2019
Line 33:		Line 33:

	There are similar projects like this, e.g. https://github.com/schollz/readable, so you’ll need to do some research into whether such a thing exists already before submitting a proposal. Maybe an existing tool could be improved instead of redoing everything from scratch?		There are similar projects like this, e.g. https://github.com/schollz/readable, so you’ll need to do some research into whether such a thing exists already before submitting a proposal. Maybe an existing tool could be improved instead of redoing everything from scratch?




	==Write a spec to a new generation of our API==		==Write a spec to a new generation of our API==
Line 44:		Line 47:
	*Rewrite a API spec using a chosen tool (e.g. OpenAPI) using best RESTful practices		*Rewrite a API spec using a chosen tool (e.g. OpenAPI) using best RESTful practices
	*Set up API demo (e.g. using Swagger UI)		*Set up API demo (e.g. using Swagger UI)



Line 52:		Line 56:
	===Development Tasks===		===Development Tasks===
	* Rewrite sitemap parser to yield found sitemap links instead of returning them to conserve memory and make results usable faster		* Rewrite sitemap parser to yield found sitemap links instead of returning them to conserve memory and make results usable faster



Line 68:		Line 73:
	=== Development Tasks ===		=== Development Tasks ===
	* Update the module in such a way that it tries common sitemap locations independently from robots.txt		* Update the module in such a way that it tries common sitemap locations independently from robots.txt



Line 76:		Line 82:
	=== Development Tasks ===		=== Development Tasks ===
	*Add RSS / Atom support to Ultimate Sitemap Parser		*Add RSS / Atom support to Ultimate Sitemap Parser



	==Build a tool to do some cool visualizations==		==Build a tool to do some cool visualizations==
Line 84:		Line 92:
	**Figure out what you'd like to visualise and how are you going to do it		**Figure out what you'd like to visualise and how are you going to do it
	**Use Gephi, a tool of your choice, or create your very own tool to implement your visualisation		**Use Gephi, a tool of your choice, or create your very own tool to implement your visualisation



	==Create PostgreSQL-based job queue==		==Create PostgreSQL-based job queue==
Line 104:		Line 114:
	**Doesn't catch fire with tens of millions of queued jobs		**Doesn't catch fire with tens of millions of queued jobs
	*(Bonus points) Actually implement the queue! If you don't get to doing this over the summer, it's fine, we would be happy with a proven spec.		*(Bonus points) Actually implement the queue! If you don't get to doing this over the summer, it's fine, we would be happy with a proven spec.



Line 111:		Line 122:
	===Development Tasks===		===Development Tasks===
	Develop a proof of concept (un)supervised ML tool for detecting subtopics of a chosen subject ("topic").		Develop a proof of concept (un)supervised ML tool for detecting subtopics of a chosen subject ("topic").



	==Do your own freehand project==		==Do your own freehand project==
Line 118:		Line 131:
	Left as an exercise to the student.		Left as an exercise to the student.

	===Skill Requirements for Potential Candidates===

			=Skill Requirements for Potential Candidates=
	*Working knowledge of Perl or Python		*Working knowledge of Perl or Python
	*Familiarity with relational databases, preferably PostgreSQL		*Familiarity with relational databases, preferably PostgreSQL
Line 128:		Line 143:
	**Craft us an email with a smart question or two,		**Craft us an email with a smart question or two,
	**Try our our tools (see https://mediacloud.org/),		**Try our our tools (see https://mediacloud.org/),
	**Run Media Cloud yourself and collect some news articles (see https://github.com/berkmancenter/mediacloud/blob/master/doc/vagrant.markdown),
	**Sign up and check out our API (see https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/api_2_0_spec.md, https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md, and the API client at https://pypi.python.org/pypi/mediacloud/).

	~~==Requirements==~~
	*Working knowledge of Perl or Python
	*Familiarity with relational databases, preferably PostgreSQL
	*Some pedantism
	*Willingness to propose, debate and object to ideas
	*Keen to work with us on writing your GSoC project proposal, as opposed to just submitting a long shot without any feedback and hoping for the best
	*Shown effort into learning what Media Cloud is all about, e.g.:
	**Make a pull request to our main code repository; (https://github.com/berkmancenter/mediacloud),
	**Craft us an email with a smart question or two,
	**Try our our tools (see https://dashboard.mediacloud.org/#demo, https://sources.mediacloud.org/, http://globe.mediameter.org/, http://focus.mediameter.org/)
	**Run Media Cloud yourself and collect some news articles (see https://github.com/berkmancenter/mediacloud/blob/master/doc/vagrant.markdown),		**Run Media Cloud yourself and collect some news articles (see https://github.com/berkmancenter/mediacloud/blob/master/doc/vagrant.markdown),
	**Sign up and check out our API (see https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/api_2_0_spec.md, https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md, and the API client at https://pypi.python.org/pypi/mediacloud/).		**Sign up and check out our API (see https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/api_2_0_spec.md, https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md, and the API client at https://pypi.python.org/pypi/mediacloud/).

BerkmanSysop: /* Projects */

2019-04-02T18:32:20Z

Projects

Show changes

BerkmanSysop: /* =Make Ultimate Sitemap Parser use asyncio */

2019-04-02T18:27:54Z

=Make Ultimate Sitemap Parser use asyncio

← Older revision		Revision as of 18:27, 2 April 2019
Line 54:		Line 54:


	====Make Ultimate Sitemap Parser use asyncio===		====Make Ultimate Sitemap Parser use asyncio====
	'''Problem Statement'''		'''Problem Statement'''
	Ultimate Sitemap Parser (https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser) is our Python module that we use to fetch trees of website sitemaps (https://www.sitemaps.org/). In production, fetching and parsing XML sitemaps it’s mostly a CPU-intensive operation as the most time gets spent on gunzipping sitemaps, parsing XML and creating objects out of them, but my guess is that the sitemap parser could be made 10-20% faster by doing I/O (namely the fetching part) asynchronously.		Ultimate Sitemap Parser (https://github.com/berkmancenter/mediacloud-ultimate_sitemap_parser) is our Python module that we use to fetch trees of website sitemaps (https://www.sitemaps.org/). In production, fetching and parsing XML sitemaps it’s mostly a CPU-intensive operation as the most time gets spent on gunzipping sitemaps, parsing XML and creating objects out of them, but my guess is that the sitemap parser could be made 10-20% faster by doing I/O (namely the fetching part) asynchronously.
Line 117:		Line 117:
	=====Development Tasks. =====		=====Development Tasks. =====
	Left as an exercise to the student.		Left as an exercise to the student.


	===Skill Requirements for Potential Candidates===		===Skill Requirements for Potential Candidates===

BerkmanSysop: /* Projects */

2019-04-02T18:27:33Z

Projects

@@ Line 33: / Line 33: @@
 There are similar projects like this, e.g. https://github.com/schollz/readable, so you’ll need to do some research into whether such a thing exists already before submitting a proposal. Maybe an existing tool could be improved instead of redoing everything from scratch?
 ====Build a tool to do some cool visualizations====
-''Problem Statement''. Since 2008, we have collected more than a half billion news articles that we have post-processed and indexed. We know quite a lot about them -- which news articles were the most linked to from other similar articles, the most and least popular / influential articles (based on shares on Facebook, tweet count, or clicks on an article's Bit.ly shortened link), specific language and terms used to describe the subject matter in each of the articles, etc., and there's a lot of potential to learn much more. Can you use your design and coding skills to help us out in visualising some of this data, e.g. create a cool network map visualization tool?
+''Problem Statement'' Since 2008, we have collected more than a half billion news articles that we have post-processed and indexed. We know quite a lot about them -- which news articles were the most linked to from other similar articles, the most and least popular / influential articles (based on shares on Facebook, tweet count, or clicks on an article's Bit.ly shortened link), specific language and terms used to describe the subject matter in each of the articles, etc., and there's a lot of potential to learn much more. Can you use your design and coding skills to help us out in visualising some of this data, e.g. create a cool network map visualization tool?
 =====Development Tasks=====
@@ Line 41: / Line 84: @@
 **Figure out what you'd like to visualise and how are you going to do it
 **Use Gephi, a tool of your choice, or create your very own tool to implement your visualisation
 ====Create PostgreSQL-based job queue====
@@ Line 75: / Line 117: @@
 =====Development Tasks. =====
 Left as an exercise to the student.
 ==Requirements==

BerkmanSysop: /* Projects */

2019-04-02T18:16:10Z

Projects

← Older revision		Revision as of 18:16, 2 April 2019
Line 18:		Line 18:

	===Projects===		===Projects===

			====Create a self-contained, browser-based page HTML -> article HTML extractor====
			'''Problem Statement'''
			For every fetched news article, we have to figure out which part of the page HTML contains the article body itself. We currently use readability-lxml (https://github.com/buriy/python-readability) for that task. However, readability-lxml is aging fast and is not necessarily still the best library around to extract body of the article from the HTML page. Also, more and more articles get loaded using JavaScript due to an ongoing "frontend everywhere!" frenzy, and our Python extractor doesn’t execute or support JavaScript. Lastly, various CDNs, e.g. Cloudflare, are blocking our crawler just because our user agent doesn't have JavaScript enabled.

			I think inevitably we'll have to switch to running a headless browser, loading each and every downloaded story in it, and then applying a well-supported third-party library, e.g. Mozilla's Readability (https://github.com/mozilla/readability), to extract article title, author and body.

			=====Development Tasks=====

			*Set up a chromeless browser
			*Set up Readability
			*Develop a HTTP service that accepts a parameter URL (and/or HTML body), loads it in the browser, runs Readability's magic, and returns the extracted HTML back to the requester.
			*Package everything in a nice Docker image

			There are similar projects like this, e.g. https://github.com/schollz/readable, so you’ll need to do some research into whether such a thing exists already before submitting a proposal. Maybe an existing tool could be improved instead of redoing everything from scratch?

	====Build a tool to do some cool visualizations====		====Build a tool to do some cool visualizations====

BerkmanSysop at 18:11, 2 April 2019

2019-04-02T18:11:58Z

← Older revision		Revision as of 18:11, 2 April 2019
Line 1:		Line 1:
	Media Cloud is an open source platform for studying media ecosystems.		[https://mediacloud.org/ Media Cloud] is an open source platform for studying media ecosystems. Media Cloud is joint project between [https://cyber.harvard.edu The Berkman Klein Center for Internet and Society at Harvard University] and [https://civic.mit.edu The Center for Civic Media] at MIT's Media Lab.

	By tracking hundreds of millions of stories published online or broadcast via television, our suite of tools allows researchers to track how stories and ideas spread through media, and how different corners of the media ecosystem report on stories.		By tracking hundreds of millions of stories published online or broadcast via television, our suite of tools allows researchers to track how stories and ideas spread through media, and how different corners of the media ecosystem report on stories.
Line 15:		Line 15:
	Project on GitHub: https://github.com/berkmancenter/mediacloud		Project on GitHub: https://github.com/berkmancenter/mediacloud

	Project Mentors: ~~Linas Valiukas [~~[mailto:linas@media.mit.edu]], ~~Hal Roberts [~~[mailto:hroberts@cyber.law.harvard.edu]]		Project Mentors: [mailto:linas@media.mit.edu Linas Valiukas], [mailto:hroberts@cyber.law.harvard.edu Hal Roberts]

	===Projects===		===Projects===

BerkmanSysop at 18:18, 27 February 2019

2019-02-27T18:18:19Z

← Older revision		Revision as of 18:18, 27 February 2019
Line 48:		Line 48:
	*(Bonus points) Actually implement the queue! If you don't get to doing this over the summer, it's fine, we would be happy with a proven spec.		*(Bonus points) Actually implement the queue! If you don't get to doing this over the summer, it's fine, we would be happy with a proven spec.


	~~====Containerize Media Cloud using Docker / Kubernetes / Mesos====~~
	Problem Description. Right now, we deploy our code using a rather archaic "SSH into server(s), type in some funky commands, press Return, pray" method. We've heard that oh-so-fancy tools like Docker allow one to containerize applications, thus introducing some nice opportunities to speed up deployments, isolate apps and resources, etc. We need to be shown how this might improve our life, and if we buy it, we'd like you to do it over the summer for us.

	~~=====Development Tasks:=====~~
	*Convince us that we need Docker (or Kubernetes, or Mesos, or whatever).
	*Come up with a plan on how to migrate live production environment to the chosen containerization tool:
	**Which Media Cloud processes (crawler, extractor, database, etc.) should run in separate containers
	**Prerequisites
	**Timeline
	**Something else?
	*Prepare for running Media Cloud in a container:
	**Split Media Cloud codebase into separately deployable containers
	**Develop configuration, tooling, monitoring and documentation needed for deploying and maintaining containers
	**Something else?

	====Implement a method to detect subtopics of a topic====		====Implement a method to detect subtopics of a topic====