The menace of link rot

I have been writing a blog on financial markets for nearly a dozen years now, and it is distressing to find that when I look up one of my older posts on that blog, many of the links in that post no longer work. This phenomenon is known as link rot.

Recently, there was a reorganization of our Institute web site and we had to set up a fair amount of URL redirection and rewriting to keep the links working. To check that everything was fine on my webpage, I ran LinkChecker to check that all internal links are working. Then out of curiosity, I ran a check on external links as well on my blog year by year:

  linkchecker --check-extern -r 1 --no-warnings --no-status URLyyyy

where URLyyyy is the url that displays all blog posts for year yyyy. What I found was that there were no broken links in the blog posts of 2016 and only one broken link in 2015. Beyond that, there were 10-20 broken links in most years (as many as 25-35 broken links in some years) out of about 300-500 links each year. This means that about 5% of all links break after about 2 years.

I could not see any pattern in the broken links. Links to government reports and court judgements from the official websites were broken presumably because they reorganized their websites and changed the URLs. Working papers and other academic papers from academic websites were broken too which makes me grateful to SSRN and ArXiv for providing permanent URLs for academic papers. Some links to media sites were broken because they went behind a paywall. I found many large financial institutions and exchanges in my list of broken links. It makes me wonder whether URL redirection and rewriting is so hard in practice for large organizations that spend millions of dollars on their websites.

What can we do about this? One solution is the Internet Archive Wayback Machine. I was happy to find that I could access some of my broken links using the Wayback Machine; at least, it confirmed that the link was correct at the time of the blog entry. Another solution that I have adopted (not as systematically as I would like) is to convert the web pages to PDF and save them locally when I link to them. This way, at least I would not lose access to these pages when link rot sets in. Unfortunately, that does not help my readers.


Predicting human behaviour is legal, predicting machines is not?

I read this Wired story about some hackers being sent to jail for “hacking” slot machines in US casinos. “Hacking” is probably the wrong word to use for this: they made money by predicting what the slot machine would do by observing it carefully, and using their knowledge of the insecure random number generator used in the software of the slot machines. It appears therefore that it is illegal to predict what a machine would do by figuring out its vulnerabilities and observing its behaviour.

The irony of the matter is that the entire business model of the casinos is built on figuring out the vulnerabilities of the human customers, predicting how they would bet under different situations and designing every minute detail of the casino to exploit these vulnerabilities. The New Yorker had a story five years ago about how a casino was redesigned completely when the customer profile changed from predominantly older male customers to more women:

So Thomas redesigned the room. He created a wall of windows to flood the slot machines with natural light. He threw out the old furniture, replacing it with a palette that he called “garden conservatory” … There are Italian marbles … Bowls of floating orchids are set on tables; stone mosaics frame the walkway; the ceiling is a quilt of gold mirrors. Thomas even bought a collection of antique lotus-flower sculptures

Casinos “monitor the earnings of the gaming machines and tables. If a space isn’t bringing in the expected revenue, then Thomas is often put to work.” The design is optimized using a massive amount of research which can justifiably be called “hacking” the human brain. If you look at the Google Scholar search results for the papers of just one top academic (Karen Finlay) in the field of casino design, you will see that she has studied every conceivable design element to determine what can cause people to bet more:

  • A comparison of ambient casino sound and music: Effects on dissociation and on perceptions of elapsed time while playing slot machines
  • Casino decor effects on gambling emotions and intentions
  • Assessing the contribution of gambling venue design elements to problem gambling behaviour
  • The Influence of Casino Architecture and Structure on Problem Gambling Behaviour
  • Measuring the Effects of Pictorial and Text Messages on Memory and Gambling Intentions Within a Casino Environment
  • The Effect of Visual Stimuli in Casinos on Emotional Responses and Problem Gambling Behavior
  • The Effect of Match and Mismatch Between Trait and State Emotion on At-Risk Gambling
  • Effects of slot machine characteristics on problem gambling behaviour

The more recent studies on human behaviour are done using a panoscope which:

features networked immersive displays where individuals are absorbed in an environment (12 feet in diameter) that surrounds them on a 360-degree basis. … Use of these panels creates a totally immersive life-like experience and facilitates the delivery of these manipulations. (Finlay-Gough, Karen, et al. “The Influence of Casino Architecture and Structure on Problem Gambling Behaviour: An Examination Using Virtual Reality Technology.” ECRM2015-Proceedings of the 14th European Conference on Research Methods 2015: ECRM 2015. Academic Conferences Limited, 2015.)

I do not see how this kind of attempt to fathom the workings of the human mind is much different from the hackers buying scrapped slot machines and figuring out how they work.

The better way to think about what is going on is to view it as a bad case of regulatory capture. The Wired story says that “Government regulators, such as the Missouri Gaming Commission, vet the integrity of each algorithm before casinos can deploy it.” The sensible thing to do is for the regulators to decertify these algorithms because the random number generators are not secure and force the casinos to use cryptographically secure random number generators. The casinos do not want to spend the money to change these slot machines and the captured regulators let them run these machines, while taxpayer money is expended chasing the hackers.

Perhaps, we should be less worried about what the hackers have done than about what the casinos are doing. Unlike the vulnerabilities in the slot machines, the vulnerabilities in the human brain cannot be fixed by a software update. Yet hacking the human brain is apparently completely legal, and it is not only the casinos which are doing this. Probably half of the finance industry is based on the same principles.

Setting up a Raspberry Pi as a home file server

Since tastes and needs differ widely within our family, the laptops and devices at my home run Linux, Windows 10, Android and iOS. Sharing data between devices running diverse operating systems is a huge pain. We often end up using email, Dropbox or Google Drive, though it is utterly silly to move a file half way round the world to share it with somebody within arm’s reach.

This is where the Raspberry Pi comes in. It is cheap enough to be within the budget of almost anybody who runs a WiFi network at home, and it is perfectly capable of running a file server accessible from Windows, Linux, Android and iOS. This post describes how I set it up for this purpose using Arch Linux. The Raspberry Pi (RPi for short) can run many different operating systems, but I chose Arch Linux ARM because it is quite powerful and also because I run Arch Linux on my laptop.


  • RPi starter kit. At a minimum, we need:
    • Raspberry Pi 3 (If you use a RPi2, you will need a USB WiFi adapter)
    • Micro USB Power Supply
    • 16 GB SD card (8 GB might be adequate) and
    • LAN cable.

      Neither a keyboard nor a display are needed as the RPi will run headless.

  • An external USB hard disk of adequate capacity (say 1TB or more) with an independent power supply. Alternatively, a powered USB hub can be used to connect a USB hard disk without independent power supply.
  • A WiFi router (administrative rights are needed)
  • A computer running Linux to setup the SD card
  • Optionally, a music system and an unused mobile phone to create a networked music system.

Continue Reading