PostRank

Louis Gray was kind enough to write a post about our new site when we launched in late October, and, as is always welcome, it was a great source of exposure and traffic for us.

One of the comments added to that post was from Mr. Gentry, who isn’t really a fan of ours. Which is just fine — some of the best feedback comes from those who aren’t convinced of the usefulness products and services. They don’t sugar coat what they don’t like or what’s not working, and they tend to provide excellently detailed information, especially if they’re developers. This makes our devs happy, as it provides concrete issues to tackle, and fodder for in-depth discussions.

I left Mr. Gentry a comment on that post, and recently he followed up again. As his comment addresses a number of issues I’ve heard in the past, and in the interest of not hijacking Louis’ blog, I’ve responded to Mr. Gentry’s comment here. Hopefully we’ll be able to clear up misconceptions, address issues, answer questions, and provide a resource of future use to others with similar questions and problems.

As always, if you’re among those with questions or problems, give me a holler any time. (Email, Get Satisfaction, or Twitter.

DGentry wrote:

I debated whether to post this publicly. Since I started it in public, I’ve decided to continue it in public.

Not a problem there. As a side note, Mr. Gentry did contact Louis regarding whether or not to continue the comment thread, but he didn’t receive a reply (I imagine his inbox is regularly mountainous), and so decided to keep the thread all in one place. Which is fine with us — can’t share information very well with others when no one can see it.

The main reason I think PostRank is not helpful to small blogs is that in order to save on bandwidth costs the PostRank metrics are collected for just a short time, and then infrequently thereafter. If you don’t have a mass of readers who descend on a new article to comment, share, and save it almost immediately, the PostRank will remain very low until AideRSS gets around to polling it again.

Mr. Gentry is correct that we don’t collect metrics on posts indefinitely. In analysis we’ve done, there is a common pattern to most posts in terms of when engagement starts to appear after a post has been published, and for how long new engagement items can be expected to keep being added. It forms a fairly consistent curve, and since you can’t account for every pattern in how audiences will engage with a post, it requires making choices that take as many as possible scenarios into account. But alas, you never get every one.

Occasionally, as noted, a post will garner some new engagement items after analysis is no longer active, and in those cases PostRank scores will end up off by a bit (in my experience, usually less than one PostRank point). When this is reported to us, we update manually.

Interestingly, Mr. Gentry’s comments did spark an interesting internal discussion about a modified way of checking for engagement items that would take these later peaks in engagement into account. No ETA on when that might happen yet, though.

I’ll have an article get comments through the course of its first day, but AideRSS shows 0 comments until some number of days later. Its a truism with RSS: people will either read an article when it first appears in their reader, or not at all. AideRSS consistently assigns my articles a minimal PostRank for the first several days, so anyone using PostRank in their reader is more likely to skip everything I write.

This is a different issue. Occasionally our analysis spiders have gotten backed up, which has led to unfortunate delays in detecting new posts and, as Mr. Gentry noted, analyzing engagement items like comments. We do monitor this, in addition to hearing from users. I am happy to report, however, that both of these issues have been addressed recently, and folks should be seeing much improved analysis times. (Of course, if you still notice delays, please tell us so we can investigate and fix them.)

I emailed AideRSS about this, and received an acknowledgement that rate limiting is done (and thats it). AideRSS needs a ping service, not just for RSS updates but to have the bot go re-check the metrics.

Ilya’s response on this one: “Most of our metrics gathering is now done via the ‘ping’ protocol he’s described. Only comments and Google trackbacks are still in polling mode.”

A secondary problem is that the PostRank metrics, most notably comment counts, suffer occasional (and large) errors. Larger blogs with larger metrics suffer little from the noise. For a smaller blog the noise will exceed the signal: articles achieve high PostRank based not on any quality of the article, but solely on measurement inaccuracy.

Yep, comment counts occasionally go haywire, though, understandably, we have to rely a lot on our community to report those instances to us. In my reply comment to Mr. Gentry on Louis’ post, I asked him to let us know if he’s seen these inaccuracies (and for what site), but he hasn’t replied.

The example Mr. Gentry notes is certainly one way engagement metrics can be off, though for high traffic sites, it tends to be more likely that something’s been upgraded/changed on the site, so engagement counts are lower than they should be. For lower traffic blogs, I’ve seen both scenarios, i.e. that engagement metrics are skewed way high, or, as aforementioned, metrics aren’t getting picked up so they’re low. Either way, they’re usually fairly easy to fix (if reported).

For my blog, a normal article gets 0-1 comments while a popular article will get 3-4 comments. The counts tracked by AideRSS are frequently off by more than this. One older article of mine shows 102 comments. This sets the PostRank bar awfully high for subsequent posts.

Posts aren’t measured against past posts in perpetuity, so a post that got 102 comments would not skew PostRank scores low for every other post that got only a few comments forever. That would make the system kind of useless. We’ve accounted for the “digg/Slashdot effect” of one post getting an unusually high number of engagement items in our analysis since the beginning, so yes, a post that gets vastly more engagement will mean some succeeding posts, if they don’t get similar engagement, will have lower PostRank scores, which is how the system is supposed to work. But the system will also account for more subtle variations, like, as noted, 0-1 comments vs 3-4 comments. And, of course, there is also the weighting of the analysis in that not all engagement items are weighted the same (more information on that here).

I asked about it on the AideRSS uservoice site: the comment count problem was acknowledged but nothing was done about correcting my site nor was any other solution offered.

This is a concern. I presume you mean you asked on Get Satisfaction? I can’t find any unanswered issues there, nor your username as I’m familiar with it. Could you send me the link to your query or send me your username? As noted earlier, I didn’t receive a reply from you to my request for the URLs of feeds displaying skewed metrics on my comment on Louis’ blog post, either, so you’re of course welcome to send me those as well. (I know of your blog, but since many of our community members have multiple blogs, I can never be sure which is experiencing issues unless the URL is provided.)

Update: Mr. Gentry provided me with the Get Satisfaction URL for his issue, and it turns out the problem was that his issue was reported as a sub-issue of a larger one. When we solved the larger issue, I closed it, and his languished, forgotten, alone, and un-solved. Totally our fault there.

Much later, I noticed that some of the uservoice answers mentioned changes in the blog template which could improve the bot’s accuracy. Unfortunately I couldn’t find the template changes mentioned, and I’m sorry to say by that point I’d given up on AideRSS and didn’t bother to follow up.

Not sure what you mean by this one. We’ve had a few issues where community members have implemented custom-built blog templates, and in some cases that’s prevented us from detecting and analyzing metrics until some tweaks have been made on our end, their end, or both. We can analyze all the major existing “pre-fab” templates, and have made upgrades to accommodate for many “home-grown” options as well. In any case, if you’re referring to something completely different, please let me know.

I was enthused about AideRSS when I first saw Louis Gray’s article about it. I would zealously check your site every time I posted an article… and was consistently disappointed. I always tweet, save to delicious, and Google Reader share every article I post. Those counts should have always been at least 1, but AideRSS consistently showed them as zero and assigned a PostRank of 1.0. Days later, the PostRank would go to somewhere between 3.0 and 6.0 (never higher, because of that one wildly inflated post being 10.0).

I think I’ve addressed these points above. In any case, you are quite welcome to contact me any time you have questions or see something that doesn’t look right with your metrics. (I have one main developer who works with me on troubleshooting, which tends to help streamline the process of sorting these things out.) I’ve asked him to take a look at your blog’s metrics to ensure everything’s working correctly there.

As noted, though, it’s not really possible for us to have a secondary system that can detect any time anything might be off with any of the hundreds of thousands of blogs in our system, so we still rely on user feedback.

According to feedburner that one wildly inflated article has the most fetches of anything I’ve written. PostRank does have an impact. However that one inflated article isn’t the article I would want to nudge new readers towards. Unfortunately I don’t get to make that choice. PostRank does.

Certainly a post that ranks as a 10 will always be a 10, so it will show up in your post history as such, and will come up in the widget as a top post (until it’s replaced by other, newer posts of equally high rank). If a post is ranked a 10 and shouldn’t be, as noted before, it should be a relatively easy thing to fix. If it’s a genuine 10 based on a lot of metrics, only way to draw attention to other posts is to have other posts that rank equally high. Of course, if I knew the exact formula for accomplishing that, I’d probably be a lot more internet famous than I am.