champagne anarchist | armchair activist

Python

Spamming after all? Revisiting the repost ratios of Vox, Upshot and 538

Recently I wrote about people who share their URLs on Twitter, and then post them again, hoping to draw even more people to their site. I said that FiveThirtyEight reposts its URLs on average 0.3 times. I was wrong: it reposts its URLs far more often. And so do voxdotcom and UpshotNYT, who didn’t even make the top 5 in my original analysis. The Upshot reposts its URLs on average as many as 0.8 times.

The reason I underestimated the repost ratios in my original analysis has to do with the fact that tweets tend to contain shortened URLs. http://nyti.ms/1rFwue2 and http://nyti.ms/1iIujpo look like different URLs. However, they point to the same article, so one should be treated as a repost of the other (or perhaps both are a repost of yet another one, who knows). If you don’t take this into account and treat them as different URLs, you’ll underestimate the number of reposts (red bar in the graph).

It’s not that I wasn’t aware of this problem when I did the first analysis. I first tried to account for this by looking up the non-shortened URLs, using the Python urllib2 module. It turned out this was very time-consuming, which was a problem since I wanted to look up quite a few URLs. Pragmatically, I decided instead to use the ‘expanded URL’ provided by the Twitter API. This method does yield higher repost ratios for 538 and the Upshot (grey bars in the graph). Still, it doesn’t really solve the problem, because the expanded URL provided by the Twitter API will sometimes be yet another shortened URL. That’s the reason I still underestimated how often people recycle their content on Twitter.

When I realised the ratios I had originally calculated were still rather low given how many reposts there appeared to be in my timeline, I decided to recalculate repost ratios using urllib2 after all. Because this method is so time-consuming, I did this for just three accounts: Vox, 538 and Upshot NYT. This resulted in repost ratios that are substantially higher (light blue bars in the graph). The new Python script is here.

Note that the ratios are snapshots calculated on a sample of the 200 most recent tweets (that is, about one to two weeks of tweets).

Not ditching R for Python just yet

As a result of the whole controversy over using Python vs R for statistical analysis and graphs, I thought I’d switch to Python. Mostly because I think it’s more practical to use the same language for different tasks, but also because it seems easier to make decent-looking graphs with Python (I’m sure some people will thoroughly disagree). And, of course, because googling for solutions using «Python» as a search term simply works better than searching for «R».

But now Brian Caffo, Roger Peng and Jeff Leek’s Data Science Specialization Course has started on Coursera and they use R. I guess I’ll have to postpone my decision.

About those weird Netflix genres

The hippest story on Twitter right now is how Alexis Madrigal of the Atlantic discovered the 76,897 genres Netflix uses to classify its movie offering. Some examples of these weirdly specific genres include Critically-acclaimed Cerebral Independent Films; Feel-good Movies starring Elvis Presley and Coming-of-age Animal Tales.

Madrigal explains how straightforward it is to navigate all the genre pages on the Netflix website by incrementing the id in the url. But then he mentions that he retrieved the genres using «an expensive piece of software called UBot Studio that lets you easily write scripts for automating things on the web». Surely a few lines of Python code could’ve done the job? In fact, I guess you could probably extract the subgenre structure and the genre elements - region, adjectives, time period etc - with nltk and regex.

Never mind that, though. Madrigal’s article is an interesting read. Here it is if you haven’t read it yet. And here’s a critique of Netflix’s algorithms by Felix Salmon of Reuters, who argues that its recommendations are no longer about quality but about offering more of the same. You watched one Dark Political Movie from the 1980s? Then we’ll show you some more Dark Political Movies from the 1980s.

Just 13% of my Linkedin connections use buzzwords

Linkedin recently released it’s newest analysis of overused buzzwords in members’ profiles. Of course, this is just a ploy to get you to volunteer more personal details («Update your profile today!»), but never mind that.

Just in case, I checked whether any of my connections engage in overusing buzzwords. Reassuringly, the majority can’t even be bothered to fill out «summaries» and «specialties» in the first place. Those who do, seldom use the top-ten buzzwords for the Netherlands, as the table below shows.

Term Percentage Term Percentage
Verantwoordelijk 0.0 Responsible 0.5
Strategisch 1.8 Strategic 3.2
Expert 3.7 Expert 3.7
Creatief 0.9 Creative 1.8
Innovatief 0.0 Innovative 1.4
Dynamisch 0.0 Dynamic 0.5
Gedreven 0.5 Motivated 0.9
Duurzaam 0.0 Sustainable 1.4
Effectief 0.0 Effective 0.5
Analytisch 0.9 Analytic 0.5

Only two Dutch buzzwords are used by more than one percent of my connections. Interestingly, their English equivalents are slightly more prevalent. 87% of my connections are completely buzzword-free. For what it’s worth, people who use buzzwords also tend to have more connections.

Full disclosure: I may have used the word «strategic» in my own profile.

Update 21 Dec - Don’t push it, Linkedin.

Incidentally, 436,567 people, that’s less than 0.17% of all the 259m Linkedin users. Not that impressive.

Method

I used these scripts (in part adapted from Matthew Russell’s Mining the Social Web) to get the «summaries» and «specialties» of connections from the Linkedin Api and process them.

Script to look up the gender of Dutch first names


This script determines the gender of Dutch persons by looking up their first name in a database of the Meertens Institute. The database indicates how often the name occurred as a first name for men and women in 2010. If the name is used for women substantially more often than for men, the name will be interpreted as female – and vice versa.

The reason I wrote the script has to to with this article on how the performance of women professional road cyclists is improving. I wanted to check whether a similar trend is going on among amateur riders, more specifically, participants in the Gerrie Knetemann Classic (incidentally, the script would take Knetemann for a woman – it’s not foolproof). The results of the ride are available online, but pre-2012 editions lack information on the gender of participants. So that’s what the script was for.

Speed of participants in Knetemann Classic

The results of the analysis aren’t exactly clearcut. The number of women participants in the 150km ride varied from 36 to 46, or 5 to 8% of the participants whose gender could be determined (the percentage for 2013 was 6%). The (median) speed of women participants rose in 2013, and more so than for men, but this rather thin to speak of a trend.

Pages