Testing Media Source Ideology Estimates Sensitivity to Samples¶
Updated Dec. 21, 2018
We previously estimated media source ideologies using a random sample of 30k Twitter users. To test how stable our ideology estimates are, we grabbed the Twitter histories of another random sample of 30k Twitter users (from our set with ideology estimates). Here are details about the first and second samples:
Users with estimated ideologies: 2,926,841
Users after removing duplicate ideologies: 1,699,669
First Sample
------------
Users in sample: 30,000
Total tweets: 45,316,914
Tweets w/ URLs: 19,563,609
Retweets w/ URLs: 13,008,418
Original tweets w/ URLs: 6,555,488
Users w/ (re)tweets w/ URLs: 17,171
Second Sample
------------
Users in sample: 30,000
Total tweets: 54,263,754
Tweets w/ URLs: 23,432,066
Retweets w/ URLs: 15,805,171
Original tweets w/ URLs: 7,626,895
Users w/ (re)tweets w/ URLs: 17,210Summary of Results¶
Sites with estimates in both samples: 6,191
Range of site estimates across both samples: -0.85 to 2.16
Mean site score difference between samples: 0.124
Correlation between scores: 0.977
Correlation between ranks: 0.958Sites that disagree the most:
In [189]:
df.sort_values('abs_score_diff', ascending=False).head(10).loc[:,['s1_mean_sharer_ideo', 's2_mean_sharer_ideo', 'abs_score_diff', 's1_num_sharers', 's2_num_sharers']]
Out[189]:
Let's set an arbitrary cutoff at 0.3 and say any site that has a score difference above that is worth looking into. Here are the sites shared by the most users with score differences above 0.3.
In [193]:
df.query('abs_score_diff >= 0.3').sort_values('s2_num_sharers', ascending=False).head(10).loc[:,['s1_mean_sharer_ideo', 's2_mean_sharer_ideo', 'abs_score_diff', 's1_num_sharers', 's2_num_sharers']]
Out[193]:
After investigating the differences between these sites and sites with better estimates, I couldn't find an easy way of trimming out the questionable scores without also removing a lot of good scores. I think the best estimates we're going to get are just combining the two samples.
Details¶
In [1]:
%matplotlib inline
import gzip, pickle, collections, itertools
import pandas as pd
import plotly.offline as plotly
import plotly.graph_objs as go
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sbs
plotly.init_notebook_mode()
Let's load in all the data.
In [2]:
%%time
user_ideos = pd.read_csv('data/cleaned_user_ideology_estimates_20180705.csv.gz', index_col=0)
uid_to_ideo = dict(user_ideos['normed_theta'].items())
MIN_USERS = 30
s1 = pd.read_pickle('data/sample1/subdomain_to_users.pkl')
s1 = {k: v for k, v in s1.items() if len(v) >= MIN_USERS}
max_length = max(len(v) for k,v in s1.items())
s1 = pd.DataFrame.from_dict({k:[uid_to_ideo.get(u, None) for u in v] + ([np.nan] * (max_length - len(v))) for k,v in s1.items()}, dtype='float16')
subdomain_to_num_urls_s1 = pd.Series({k: len(v) for k,v in pd.read_pickle('data/sample1/subdomain_to_urls.pkl').items()})
s2 = pd.read_pickle('data/sample2/subdomain_to_users.pkl')
s2 = {k: v for k, v in s2.items() if len(v) >= MIN_USERS}
max_length = max(len(v) for k,v in s2.items())
s2 = pd.DataFrame.from_dict({k:[uid_to_ideo.get(u, None) for u in v] + ([np.nan] * (max_length - len(v))) for k,v in s2.items()}, dtype='float16')
subdomain_to_num_urls_s2 = pd.Series({k: len(v) for k,v in pd.read_pickle('data/sample2/subdomain_to_urls.pkl').items()})
In [3]:
%%time
df = pd.DataFrame({
's1_mean_sharer_ideo': s1.mean(),
's1_stddev_sharer_ideo': s1.std(),
's1_num_sharers': s1.count(),
's1_num_uniq_urls': subdomain_to_num_urls_s1,
's2_mean_sharer_ideo': s2.mean(),
's2_stddev_sharer_ideo': s2.std(),
's2_num_sharers': s2.count(),
's2_num_uniq_urls': subdomain_to_num_urls_s2,
}, index=s1.columns.intersection(s2.columns))
display(df.sample(5))
In [4]:
df.info()
Sample Comparisons¶
Now we have estimates from the second sample. Let's load our estimates from the first sample and compare them. I'm including some transformations of the data to make things more linear and easier to eyeball.
In [87]:
df['s1_ideo_rank'] = df['s1_mean_sharer_ideo'].rank()
df['s2_ideo_rank'] = df['s2_mean_sharer_ideo'].rank()
df['abs_rank_diff'] = (df['s1_ideo_rank'] - df['s2_ideo_rank']).abs()
df['abs_score_diff'] = (df['s1_mean_sharer_ideo'] - df['s2_mean_sharer_ideo']).abs()
df['s1_num_sharers_logged'] = df['s1_num_sharers'].apply(np.log)
df['s2_num_sharers_logged'] = df['s2_num_sharers'].apply(np.log)
df['s1_num_uniq_urls_logged'] = df['s1_num_uniq_urls'].apply(np.log)
df['s2_num_uniq_urls_logged'] = df['s2_num_uniq_urls'].apply(np.log)
df['abs_score_diff_sqrt'] = (df['abs_score_diff']).apply(np.sqrt)
df['abs_rank_diff_logged'] = (df['abs_rank_diff'] + 1).apply(np.log)
#df = df.drop('twitter.com')
In [88]:
f, ax = plt.subplots(figsize=(15, 10))
_ = sbs.heatmap(df.corr(), annot=True, fmt=".3f", linewidths=.5, ax=ax, vmin=-1, vmax=1, center=0)
Observations
- Correlation between the scores (Pearson) is good at 0.977 (
s1_mean_sharer_ideovs.s2_mean_sharer_ideo) - Correlation between the ranks (Spearman) is still good but less so at 0.958 (
s1_ideo_rankvs.s2_ideo_rank) - There's a bunch of correlation here, but right now I'm just going to look at correlations between the score and rank differences and features to see what might be contributing to disrepancies between samples.
abs_score_diff_sqrtis correlated with within-sample standard deviation, which makes sense- It's inversely correlated with number of sharers, which also makes sense
- It's slightly inversely correlated with the number of unique URLs, but that's partially dependent on number of sharers, so duh
- Score difference is inversely correlated with number of sharers, which also makes sense
- I don't fully grasp the difference between rank and score, and why they correlate so differently with things. I think I need more graphs to do that.
In [79]:
_ = df['s1_mean_sharer_ideo'].plot.hist(bins=200, alpha=0.4)
_ = df['s2_mean_sharer_ideo'].plot.hist(bins=200, alpha=0.4)
This look good. Let's graph every pair of variables.
In [90]:
_ = sbs.pairplot(df, vars=['s1_mean_sharer_ideo', 's2_mean_sharer_ideo', 's1_ideo_rank', 's2_ideo_rank',
's2_num_sharers_logged', 's2_stddev_sharer_ideo', 's2_num_uniq_urls_logged',
'abs_score_diff_sqrt', 'abs_rank_diff_logged'],
plot_kws={'alpha': 0.1}, diag_kind='kde')
Observations
- The mean-mean graph and the rank-rank graph look decent. Most of the discrepancies are sites in the middle, which can also be seen in the mean_sharer_ideo vs. abs_score_diff_logged graph.
- The graph of mean sharer ideology vs. standard deviation of sharer ideologies looks like the fermata symbol. The curve makes sense because the ideology distribution is bimodal, so the more mixed those two modes are, the higher the standard deviation. The dot in the fermata is a cluster of Spanish language sites which have users that are actually estimated to be center-right, hence the lower standard deviation.
The range of scores for media sources is -0.84 to 2.0, so a change of 0.3 is close to the max we'd want here to trust our scores. What are the biggest sites with a score difference greater than 0.3?
In [102]:
df.query('abs_score_diff >= 0.3').sort_values(by='s1_num_sharers', ascending=False).head(30)
Out[102]:
Observations
- There's a bunch of Florida stuff in here. (
floridapolitics.com,naplesnews.com,floridadisaster.org,dos.myflorida.com,wesh.com) - There's a bunch of federal gov't stuff in here.
- Fox 8 is Cleveland
- WMAL is DC AM radio
- KOB is New Mexico
- WBRC is Alabama
- Fox 43 is Pennsylvania
- A few Texas
- So it looks like a lot of swing states are represented here, but not all local media are swing states.
Let's see what the underlying user ideology distributions look like for each sample for these sites.
In [175]:
fig, axes = plt.subplots(5, 6, figsize=(20, 12), sharex=True)
subdomains = df.query('abs_score_diff >= 0.3').sort_values(by='s1_num_sharers', ascending=False).head(30).index.values
for i, s in enumerate(subdomains):
ax = axes[i // 6][i % 6]
ax.set_title('{} - {:.2f}'.format(s, df.loc[s, 'abs_score_diff']))
s1[s].plot.hist(bins=30, alpha=0.4, density=True, ax=ax)
s2[s].plot.hist(bins=30, alpha=0.4, density=True, ax=ax)
Observations
- Some of these look really different between samples. Sites like
floridapolitics.com,naplesnews.com,ch7.io - If these underlying distributions are so different between samples, I'm not sure what there is we can do. Just crazy bad samples?
What happens if I bootstrap within each sample for a site to see what confidence intervals for the mean might look like?
In [196]:
subdomain = 'floridapolitics.com'
_ = pd.Series([s1[subdomain].dropna().sample(frac=1, replace=True).mean() for _ in range(1000)]).plot.hist(bins=50, alpha=0.4)
_ = pd.Series([s2[subdomain].dropna().sample(frac=1, replace=True).mean() for _ in range(1000)]).plot.hist(bins=50, alpha=0.4)
Observations
- Those are no where close.
I don't know what to do with this, so here's an interactive graph.
In [199]:
interactive_scatter(df, 's1_mean_sharer_ideo', 's2_mean_sharer_ideo')
In [198]:
def interactive_scatter(data, x, y):
scatter1 = go.Scatter(
x=data[x],
y=data[y],
mode='markers',
text=data.index,
hoverinfo='text',
marker=dict(
#color=site_ideo['mean_ideology'].loc[dom_ideo_normed_hist.index],
#cmin=-1, cmax=1,
#size=plotted_df['num_uniq_users_sample1'],
#sizemode='area',
#sizemin=2,
#sizeref=30,
opacity=0.1
)
)
layout1 = go.Layout(
#title ='Domain Similarity by Distribution of Audience Ideology',
hovermode = 'closest',
#xaxis = dict(visible = False),
#yaxis = dict(visible = False),
width = 800,
height = 600
)
plotly.iplot(go.Figure(data=[scatter1], layout=layout1))