Category Archives: web

Download from Tivo fails with HTTP error 400

So tonight I was wondering why no new shows had been downloaded from my Tivo to my file server. Investigating my home-grown script’s log output, I  could see that all download attempts were met with an HTTP error 400 since Sunday. (My script works similarly to kmttg.)

I immediately tried in Safari, and had no problems downloading one of these shows. So more debugging was necessary. Enabling additional output for Python’s urllib2 (used for downloading the XML table of contents) and curl (used for downloading the media files) showed an unusual Set-cookie response header on all of the requests, setting a new value for the “sid” cookie. Quick recap: when you load the XML TOC over HTTPS, the Tivo sets a “sid” cookie which you need to supply to download the actual show’s file. No problem, my script was recording the cookie using cookielib.CookieJar, passing the resulting file to curl. Or so I thought. Inspecting the cookie jar I found it to be empty. How could that be?

At this point I turned to Firefox and the Live HTTP Headers extension. Interestingly, the problem was the same in Firefox, so it wasn’t a problem in my script. but was was going wrong? After some experimentation with variations of the Set-cookie header (using my internal web server and Apache’s very useful mod_asis), plus remembering the details of the cookie protocol, it became clear:

Set-Cookie: sid=542329DF12A3586B; path=/; expires="Saturday, 16-Feb-2013 00:00:00 GMT";

If a cookie has an expires attribute, and the current date and time is later than the date specified, the cookie becomes invalid, and the HTTP client should remove it and not transmit it anymore. If a server sends a Set-Cookie header with an expires attribute in the past, the cookie should be deleted. Bingo! (Safari  had an old cookie still set, so the Tivo just used that and did not remove the cookie.)

So the Tivo is apparently trying to set a persistent cookie (confusingly named “sid” as it’s not a session cookie), but doing so with a date in the past. And everything was working just fine until last Saturday! So either Tivo pushed a software update on Sunday, changing the expires date of that cookie, or it has had that date for ever, and eternity just ran out. I haven’t reverse engineered the firmware, but I would bet that someone way back when Tivo got started hard-coded the date “way in the future”, and everbody promptly forgot about it.

No word from Tivo yet, but this Tivo forum thread mentions the same broken cookie.

I managed to add a hack to my script that extracts the “sid” from the Set-Cookie header manually and hand the proper cookie value to curl for the downloads. And I got to learn about mechanize for Python!

Keep those privacy-eating websites at bay

Facebook, Google, Twitter, and all those other services that sell ads want to know as much about you as they can possibly find out. Consider for example Facebook’s Like button that is part of most web pages: even if you don’t click it, Facebook knows that you have viewed that page. How can they know? Since you have logged in to Facebook before, they have stored a cookie in your browser. And the Like button’s image is loaded from Facebook’s servers, so that cookie is sent to Facebook, together with information about the page that you’ve just visited. (My blog does not contain any tracking codes, not even Google analytics.)

So how can you avoid this? You could delete all stored cookies every single time you log out of Facebook (and Twitter and Google and…) but that’s not really convenient. Fluid is a small browser for Mac OS X (based on the same WebKit engine as Safari), but with a twist: instead of using the application to browse the Internet, you create specialized browsers for specific websites. I’ve set up a number of them: one each for my personal Facebook and Twitter accounts plus a couple more for accounts that I help maintain for my historic railway club.

Since these specialized browsers all store their cookies seperately from each other, I can use my main browser without ever having a Facebook or Twitter login. For those services, I appear as some random surfer, not connected to my actual profile.

Of course, this little trick is not perfect. WIth advanced analytics that all the ad networks employ, some information is still gathered about me (such as when I click a link in my Facebook browser that takes me to a different website), but I still feel a lot better about not giving away all my browsing history all the time.

Services to sign up for: app.net and ifttt.com

So I’ve plunked down 100 bucks for an early bird developer account at join.app.net, a new social network. So why would anyone pay for something like this, if Facebook and Twitter and everything else is free? Because it isn’t.

Facebook and Twitter live off selling their users to advertisers. Nothing wrong with that, but it means that more and more, they choose to implement features and control the user experience to maximize value for the advertisers. What the users want and how they would like to be treated becomes less important. And third-party applications and their developers make things complicated, since they do not help generating ad revenue, so more and more, they’re shut out or severely restricted.

Dalton Caldwell was annoyed by that, decided to do something about it: create a new social media platform, but instead of financing it through advertising, have the users pay for it. At least in principle, it aligns the interests of the platform company with that of the users, since happy users will generate more revenue. Right now, it’s an experiment (but it’s looking like it might get very successful, with over 12.000 paying users signing up in less than two months). Right now I have the bragging rights to be the first among my friends to have signed up, but that mainly means that none of my friends are on app.net yet. We’ll see if that changes within the next 12 months.

Which brings me to the second service: if-this-then-that, or IFTTT for short. I now have a plethora of social media accounts, and distributing the various thoughts that I sometimes find worth publishing can become tedious. IFTTT helps with that: it monitors your account on any of 30 or so services (you decided with ones), and then cross-posts new posts from one service to any other one of your choosing. You decide which service to monitor and what to post where by settings up simple recipes: if I publish a new blog post, take it’s title and URL and post it to app.net, Facebook and Twitter. I’ve just started using it, but from what I can see from my friends using it, it looks like a very useful tool.

Where’s the doctor now?

The oldest, still running science fiction TV series, Doctor Who, has the Doctor jumping through time and space. A lot.

Now David McCandless has collected all of them in a huge dataset on his blog, waiting for someone to visualize them. As David writes in the Guardian,

I really wanted to do a mega-visualisation of all of the Time Lord’s journeys. But faced the cosmic task of trawling through well over 200 episodes, logging every time TARDIS was hurled through time and space.

Can’t wait for the results to show up!

Panorama movie of Texas Stadium demolition

Texas Stadium, previous home of the Dallas Cowboys, was demolished on April 11th. Immersive Media recorded the demolition of the main structure using a 360° high speed video camera, so you can pan and zoom around while the stadium is collapsing around you. I don’t know if this is the first such recording, but it certainly is a nice application of panoramic photo and video recording, so that you can experience something from a position you would not want to be in personally. Too bad the camera got knocked out about halfway through the sequence.

Texas Stadium demolition

Screenshot of Immersive Media's flash viewer, displaying the demolition of Texas Stadium

via BoingBoing

Fixing Flash for Linux Firefox-3.5.2

If you have trouble with Flash segfaulting in Linux Firefox-3.5.2 (the symptom is a browser hang), I fixed that by downgrading Flash from Flash10 to Flash9. Of course Flash9 is full of security holes. In combination with the Flashblocker Firefox extension and only activating it on trusted sites it seems like a reasonable approach. Of course for all we know the newest Flash10 is full of security problems, too.

Fonts are finally coming to the web

I have to admit that I didn’t follow developments too closely for the last couple of years, but I was rather surprised today to find that Safari 4, Firefox 3.5 and Internet Explorer 6 to 8 support downloadable TrueType fonts in a compatible and useful manner.

Slashdot post, linked Slate article, nice overview page.

A couple of things that are buried in the pages linked above, but which helped me to get up to speed:

<style>
@font-face {
    font-family: "testing";
    src: url("output.ttf") format("truetype");
}
</style>
<!--[if IE]>
    <style>
    @font-face {
        font-family: "testing";
        src: url("output.eot");
    }
    </style>
<![endif]-->
<div style="font-family: testing">
Hello, Multiple Browser World!
</div>

Now I just need to quickly build a WordPress and a MediaWiki template, and we’re all set 🙂