In [1]:
import pickle
import tldextract, joypy
import pandas as pd
import seaborn as sns
%matplotlib inline
sns.set()
NewsGuard Data¶
Let's take a look at what NewsGuard has for data. A CSV of NewsGuard data joined with Facebook ideology scores and Media Cloud election retweeter scores is available here.
Getting Data¶
They didn't have any policies I could find against scraping data, so I wrote a quick scraper that tried to be nice about it. I couldn't find a way to get them to list the sites they had rated, so I put together a list of ~22,700 unique domains we had seen in some of our research. I then queried their system for each of those domains and stored all the data.
Looking at the Data¶
Their data comes back as a fairly detailed, multi-level JSON document. Here's an example document:
In [2]:
ratings = pickle.load(open('scrape_news_domains_20180827T14:08:00Z-0400.pkl', 'rb'))
ratings['nytimes.com']
Out[2]:
Observations
- Gordon and Steve get their own fields.
createdDateandupdatedDateare empty.- They have authors, editors, and reviewers on each.
- The interesting bits look like
criteria,metadata,rank, andscore. - There is an extensive source list.
- There is an
orientationmetadata field. - The metadata also contains links to social media pages for the media source
- The
criteriafield contains the attributes they judge on - The
rankfield contains their judgement.
I looked at sample pages for each unique rank code. Here are the messages a user receives for each rank:
In [3]:
rank_info = {
'TK': 'This website is still in the process of being rated by NewsGuard.',
'T': 'This website generally maintains basic standards of accuracy and accountability.',
'N': 'Proceed with caution: This website generally fails to maintain basic standards of accuracy and accountability.',
'P': 'This website publishes content from its users that it does not vet. Information from this source may not be reliable.',
'S': 'This is a satire or humor website. It is not an actual news source.'
}
They obviously didn't return ratings for all the domains I queried. Let's see how many ratings I actually retrieved, not including ratings of rank TK.
In [4]:
have_ratings = [d for d,r in ratings.items() if r['id'] is not None and r['rank'] != 'TK']
len(have_ratings)
Out[4]:
In [5]:
site_to_id = {s:r['id'] for s,r in ratings.items() if r['id'] is not None}
ratings_by_id = {r['id']:r for r in ratings.values() if r['id'] is not None}
domain_to_rating = {tldextract.extract(s).registered_domain.lower():ratings_by_id[i] for s,i in site_to_id.items()}
def get_subkey(rating, key, title):
if rating[key] is None:
return None
metadata = next((m for m in rating[key] if m['title'] == title), None)
return metadata['body'] if metadata else None
def get_metadata(rating, title):
return get_subkey(rating, 'metadata', title)
def get_criteria(rating, title):
return get_subkey(rating, 'criteria', title)
Orientation¶
The orientation data seems interesting. Let's see what that looks like.
In [6]:
orientations = {}
for domain, rating in domain_to_rating.items():
orientations[domain] = get_metadata(rating, 'orientation')
ng_orientations = pd.Series(orientations)
ng_orientations.name = 'ng_orientation'
ng_orientations.value_counts()
Out[6]:
In [7]:
ng_orientations.groupby(ng_orientations).apply(lambda g: g.sample(5))
Out[7]:
Observations
- Most of the orientations are
N/A - There are only ten
Far Left. - It looks like they've been either really conservative in assigning orientations to anything, or they stopped at some point.
How do these orientations correlate with election retweeters and Facebook ideologies?
In [8]:
mc_scores = pd.read_csv('election_retweeter_polarization_media_scores.csv')
mc_scores['domain'] = mc_scores['url'].apply(lambda u: tldextract.extract(u).registered_domain.lower())
mc_scores = mc_scores.set_index('domain')
orientation_order = ['Far Left', 'Slightly Left', 'N/A', 'Slightly Right', 'Far Right']
joined = mc_scores.join(ng_orientations).dropna()
_ = sns.catplot(x='score', y='ng_orientation',
order=orientation_order, data=joined)
Observations
- There's definitely a relationship, so that's good.
N/Ahas a wide range of scores.- There's more disagreement in the middle, which is intuitive.
How about the Facebook data?
In [9]:
facebook = pd.read_csv('facebook_ideology_estimates.csv')
facebook['domain'] = facebook['domain'].apply(lambda d: tldextract.extract(d).registered_domain.lower())
facebook = facebook.set_index('domain')
_ = sns.catplot(x='avg_align', y='ng_orientation',
order=['Far Left', 'Slightly Left', 'N/A', 'Slightly Right', 'Far Right'],
data=facebook.join(ng_orientations).dropna())
Observations
- Looks like a little bit less disagreement in the middle relative to election retweeter method.
N/Askews left.Slightly LeftandSlightly Rightlook to extend about as far to the edges as their "Far" counterparts, but just have more variance.
Rank and Score¶
Let's build a flat dataset of interesting fields so things are a bit easier to look at. Then let's look at rank, score, and criteria.
Criteria is documented here.
In [10]:
domains = {}
for domain, rating in domain_to_rating.items():
if rating['id'] is None or rating['rank'] == 'TK':
continue
domains[domain] = {
'rank': rating['rank'],
'orientation': get_metadata(rating, 'orientation'),
'score': rating['score'],
'falseContent': get_criteria(rating, 'falseContent'),
'basicStandards': get_criteria(rating, 'basicStandards'),
'newsOpinion': get_criteria(rating, 'newsOpinion'),
'deceptiveHeadlines': get_criteria(rating, 'deceptiveHeadlines'),
'accountabilityPractices': get_criteria(rating, 'accountabilityPractices'),
'ownership': get_criteria(rating, 'ownership'),
'labelsAdvertising': get_criteria(rating, 'labelsAdvertising'),
'management': get_criteria(rating, 'management'),
'contentCreators': get_criteria(rating, 'contentCreators'),
'active': rating['active'],
}
ng = pd.DataFrame.from_dict(domains, orient='index')
print(ng.shape)
ng['rank'].value_counts()
Out[10]:
In [11]:
ng[ng['rank'] == 'N']['score'].sort_values()
Out[11]:
In [12]:
_ = sns.distplot(ng['score'], kde=False)
Observations
- Most sites pass. Only 39 out of 533 fail.
- I know most of these that fail. Most of them look to be on the right.
- Most of the sites are very near 100.
In [13]:
criteria = [
"falseContent",
"basicStandards",
"newsOpinion",
"accountabilityPractices",
"deceptiveHeadlines",
"ownership",
"labelsAdvertising",
"management",
"contentCreators",
]
ng.loc[:,criteria].apply(lambda c: c.value_counts())
Out[13]:
In [14]:
_ = ng.loc[:,criteria].apply(lambda c: c.value_counts()).loc['No',].sort_values().plot.barh(title='# Sites Failing each Criteria')
Observations
- Accountability practices and management are where most get knocked down. Those are defined as "Regularly corrects or clarifies errors" and "Reveals who’s in charge, including any possible conflicts of interest".
Scores by Orientation¶
I'll look at scores by NewsGuard assigned orientation first, and then switch over to the Facebook numbers to get something more fine-grained.
In [15]:
ng = ng[(ng['rank'] == 'T') | (ng['rank'] == 'N')]
ng.groupby('orientation')['rank'].value_counts()
Out[15]:
In [16]:
_ = sns.stripplot(x='score', y='orientation', data=ng, order=orientation_order)
In [17]:
_ = sns.boxplot(x='score', y='orientation', data=ng, order=orientation_order)
Observations
- All of the orientations' scores look pretty spread out.
Slightly Lefthas the fewest outliers with low scores.- The boxplots show that scores drop off as you approach the edges of ideology.
- The
Far Righthas more of its mass in the lower scores. In fact, the median is actually failing.
Let's look at Facebook numbers.
In [18]:
import plotly.offline as plotly
import plotly.graph_objs as go
plotly.init_notebook_mode()
scatter1 = go.Scattergl(
y=ng.join(facebook)['score'],
x=ng.join(facebook)['avg_align'],
mode='markers',
text=ng.join(facebook).index,
marker=dict(
color=ng.join(facebook)['rank'].factorize()[0],
colorscale=[[0, 'rgb(200,20,20)'], [1, '#398937']]
)
)
layout1 = go.Layout(
title ='NewsGuard Scores vs Source Ideology',
hovermode = 'closest',
xaxis = dict(title = 'Ideology by Facebook Average Alignment'),
yaxis = dict(title = 'NewsGuard Score'),
)
plotly.iplot(go.Figure(data=[scatter1], layout=layout1))
Observations
- Most of the center/center-left is up near 100, but a few drop down below failing. RT is one of them.
- Scores fall off a little bit approaching the far left.
- The center-right is missing, like we've seen before.
- The far right is its own thing, and the scores are a lot more variable.
In [21]:
ng.join(facebook).join(mc_scores, rsuffix='_mc').to_csv('newsguard_facebook-ideo_election-retweet_joined.csv')