Internet Monitor

Internet Monitor is a research project based at Harvard University's Berkman Center for Internet & Society. Internet Monitor's aim is to evaluate, describe, and summarize the means, mechanisms, and extent of Internet content controls and Internet activity around the world. The project helps researchers, advocates, policymakers, and user communities understand trends in Internet health and activity through research, analysis, and data visualization.

Internet Monitor has several technical components:

Platform

Contact: Ryan Morrison~Westphal (rwestphal@cyber.law.harvard.edu) The Internet Monitor platform and Access Index (launched in July 2014) give policy makers, digital activists, researchers, and user communities an authoritative, independent, and multi-faceted set of quantitative data on the state of the global Internet. The platform brings together 15 indicators on Internet access and infrastructure to create the Internet Monitor Access Index, which offers a starting point for further analysis of Internet access conditions in 92 countries.
July 15, 2014: Welcome to Internet Monitor!
July 22, 2015: A Berkman Center summer intern explores Internet Monitor

Tech:

• Ruby on Rails
• JavaScript

Project ideas:

• Support switching between current and archived Access Indexes in both the website and data API
• Updates the indicator weight sliders feature

Dashboard

The Internet Monitor dashboard (launched in September 2015) offers users the opportunity to customize a collection of data visualization “widgets” according to their interest. The dashboard compiles and curates data from multiple sources, including primary data collected by the Berkman Center and our partners, as well as relevant secondary data. Users can create individual boards that provide a real-time view of the state of the Internet across a variety of dimensions, enable easy comparisons across countries and data sources, and are easy to configure, edit, and share.
September 28, 2015: Announcing the Internet Monitor data dashboard!

Tech:

• Meteor
• Bootstrap 3
• Project ideas:
• Measurement Lab widgets
• Data-agnostic visualization widgets

Website availability testing

Contact: Justin Clark (jclark@cyber.law.harvard.edu) Internet Monitor is developing a suite of tools to enable the automated collection of data on website availability in a number of countries worldwide. This effort has several components: Working together with Dyn, we are employing data collection tools tailored specifically for assessing website availability and gathering evidence of deliberate blocking. We are also developing global and country-specific lists of URLs to test and developing additional sources of data, including NetClerk (which uses proxies located in data centers worldwide to test website accessibility) and FarFlung (a Chrome extension that allows users to participate voluntarily in testing URLs from their browsers). We are developing an analysis platform that will ingest data from the Dyn network of virtual servers and multiple other sources—including NetClerk, FarFlung, and crowdsourced website availability reporting platform Herdict to more accurately determine the availability of different sites in different locations. We are building a real-time testing window, initially available only to invited participants, to enable on-demand testing in any of the supported locations worldwide.

Tech:

• AMQP (RabbitMQ)
• Google Chrome extension
• Ruby on Rails
• WARC and HAR formats
• Meteor
• Internet protocol stack

Project ideas:

• Planet Lab, Measurement Lab, RIPE ATLAS integrations
• Mobile component
• System-on-a-Chip (SoC) collectors (Arduino, CHiP, Raspberry Pi, ESP8266)
• Tor integration