Running LaTeX, Python and more in an Android Phone

Because you want to travel without your laptop

If you are an open source enthusiast, you have a problem when you travel without a laptop. This is one situation where you are at a serious disadvantage compared to somebody who uses Microsoft Office (Word, Excel and Powerpoint) or Google’s G Suite (Google Docs, Sheets and Slides). Even while travelling, MS Office users can use a mobile App to correct a spelling mistake in Word or Powerpoint or change some parameters in an Excel spreadsheet. If they need more serious work, they would often be able to borrow a laptop from whoever they are visiting and be almost sure that MS Office is already installed on that machine.

However, you have no such luck with correcting your LaTeX/Beamer presentation or editing your Python code. There are no mobile Apps for LaTeX. There are Apps for Python (QPython) for example, but using them without access to your other development tools can be very tortuous. Your friends might be happy to lend their laptops, but they would probably not have LaTeX and Python installed. The only solution appears to be to carry your laptop with you all the time.

But there is a better solution if you have a decent Android smartphone (say 4GB of spare storage space, reasonably powerful processor and say 2GB or more of RAM). You can turn your phone into a miniature version of your laptop by installing a desktop Linux distribution inside your Android phone and then installing all your favourite open source software inside that. It is true that every Android phone is already running Linux, but that is just the Linux kernel on top of which runs the Android operating system. What we want to do is to take the same Linux kernel and run a standard desktop Linux operating system on top of that. In other words, we want all the Linux system administration tools and our preferred application software running on Android’s Linux kernel: we want the GNU tools, LaTeX, Python and lots of other things.

Thankfully, there is an Android App called GNURoot Debian which makes all this possible: it installs a fully functional Debian (Jessie) Linux system right inside your phone. Most importantly, it does not need a rooted phone to work.

Identifying and installing the desired software

The first step is to install the Android Apps that you need. In addition to GNURoot Debian itself, you also need the Hacker Keyboard: you are going to be using the command line terminal a lot and to do that effectively you need arrow keys, control keys and function keys. The Hacker Keyboard gives you all that you want, and to my knowledge there is no other keyboard that provides all this. Just in case you are wondering, both GNURoot Debian and Hacker Keyboard are open source projects.

My work requires LaTeX and the full SciPy Stack (a collection of open source software for scientific computing in Python that includes Numpy, Scipy, Matplotlib, and Pandas). Most of my computations happen in Python and I use Pweave to include Python output into my LaTeX files automatically. In addition, as explained later, I find it useful to run a Jupyter notebook (or its older version, the ipython notebook) on the phone.

After installing GNURoot Debian, I opened the App, switched to the Hacker Keyboard, and ran the following commands in the Debian command terminal to install the Linux software that I needed. You may be interested in a different set of software, but you will observe that the standard installation methods work, and you could install your favourite software the same way.

    apt-get update
    apt-get upgrade
    dpkg-reconfigure tzdata
    apt-get install man-db nano  build-essential
    apt-get install python3  python3-scipy python3-matplotlib python3-pandas python3-jedi
    apt-get install python3-pip ipython3 ipython3-notebook 
    apt-get install texlive
    apt-get install fonts-noto emacs  pandoc pdftk aspell aspell-en

    # pip install of Pweave fails in Python 3 (http://mpastell.com/pweave/#install-and-quickstart) 
    # so we install from source
    apt-get install curl
    url=https://pypi.python.org/packages/f6/2f
    url+=/e9735b04747ae5ef29d64e0b215fb0e11f1c89826097ac17342efebbbb84
    url+=/Pweave-0.25.tar.gz#md5=d406395ee4a578c30ad498426402b8e6
    curl $url > Pweave-0.25.tar.gz
    tar -xvf Pweave-0.25.tar.gz
    cd Pweave-0.25
    python3 setup.py install
    cd ..
    # clean up 
    rm -r Pweave-0.25
    rm Pweave-0.25.tar.gz

Transferring files in and out

The Debian Linux is like an island inside the phone: the file system (/home or /usr) is all contained in the private data storage space of GNURoot Debian and is not accessible to other Apps. Nor are these folders visible to your PC when you connect your phone to it using a USB connection. So we need to build bridges from this island to other Android Apps, to our PCs and to the cloud so that we can conveniently transfer stuff into and out of it. Since folders in the internal storage of the phone are accessible to all Apps and also to the PC over a USB connection, I created a folder mydebian in internal storage using the File Manager in the phone. GNURoot Debian mounts the internal storage as /sdcard and so this new folder will appear inside Debian Linux as /sdcard/mydebian. I also created a convenient symlink /j to this folder:

    # create a short symlink to a folder accessible to other applications
    ln -s /sdcard/mydebian /j

Now I can use my preferred Android text editor App to edit LaTeX or Python source files in the mydebian folder and access them inside Debian Linux in the /j folder. Running pdflatex on a .tex file in /j will produce a PDF file in the same folder and I can use standard Android Apps to view the PDF files. The same is true of image files produced by running Python code. This setup also makes it easy to transfer files to and from the cloud or a computer. For example, I can now copy all my tex style files to mydebian/texmf using a USB connection or through the cloud. Then, in the Debian terminal, I can either copy /j/texmf to /home/texmf or make LaTeX find my style files directly from /j/texmf with the following command:

    # this did not work: tlmgr conf texmf TEXMFHOME  "/j/texmf"
    # so we do it using .bashrc
    echo "export TEXMFHOME=/j/texmf"  > ~/.bashrc

I usually copy LaTeX and Python source files from my PC to the mydebian folder on the phone through a USB connection or through the cloud. In the latter case, the files are encrypted using GPG and are decrypted on the phone using the OpenKeyChain Android App. The decrypted files are also placed in the mydebian folder in internal storage. Therefore, most of the time when I am running Debian Linux, I am in the /j folder; it plays the role that is normally played by my home folder in standard Linux. There is however one thing you cannot do in this folder: if you create a shell script or a Python file inside /j and try to make it executable in order to run it directly, you will find that it does not work. If you issue a chmod +x command, it would not produce any error message, but the file would not actually become executable. The reason is that Android mounts the storage card with the -noexec flag which denies execute permission to any file here. If you want to create an executable file, you will first have to copy them to a folder somewhere else (for example, inside /home or /usr) and then use chmod to make them executable.

Graphic Interface: notebook servers and web servers

GNuRoot Debian can also run a graphical environment (X windows) using a VNC server. But I do not use this because it is slow and is more suited for tablets than for phones. Instead, I use native Android Apps when I need a graphical environment.

For editing and running Python code, I find the ipython notebook server most useful. In the Linux terminal, I cd to a suitable folder and run the command ipython3 notebook to start the notebook server. Then I can go to the browser in the phone and point it to localhost:8888 to connect to this server. I can write Python code in this notebook and run it. Plots produced by matplotlib also display in the browser if I add the line %matplotlib inline at the beginning of the code.

Many times, even when I am travelling, I can get access to a Windows PC. Much of the software on the Windows PC is totally useless for me, but the web browser is quite valuable. The Chrome web browser has the same look and feel on Linux, Windows and Android. When I use Chrome on a Windows PC, I can forget that it is a Windows PC. What is interesting is that the browser can connect not only to the world wide web but also to the Debian Linux running on my phone provided I run a server there. Most commonly, I connect to the ipython3 notebook mentioned above. For example, I can walk into a classroom or conference hall anywhere, connect my mobile to the instructor’s or presenter’s laptop, run the browser in that laptop and connect to ipython3 notebook server in my mobile. The Python code and its output now appear on the projection screen for all to see. I do not even need to touch the mobile as everything is controlled from the laptop.

There are many ways to connect the mobile to the laptop. I can use a USB cable and use USB tethering to make the mobile accessible from the browser. Alternatively, I can create a WiFi hotspot in my phone and connect the laptop to this hotspot. In both cases, the server has to be made visible to the local network. This requires adding the switch --ip=* while starting the server: ipython3 notebook --ip=*. Also, we then need to know the IP address of the phone; the command hostname -I in the Debian terminal gives the IP address of the phone (say 192.168.1.100). Then the Chrome browser in the laptop has to be pointed to 192.168.1.100:8888. A third (more risky) way of creating the network connection can be used if the laptop is connected to a WiFi network. I can connect the phone to the same network instead of setting up a hotspot on the phone and connecting the laptop to it. Either way, the laptop and the phone need to be on the same network for the connection to be possible. Needless to say exposing the notebook server to a large WiFi network can be a serious security risk (the whole point of the IPython notebook is arbitrary code execution), and I would recommend this only under exceptional circumstances.

In some situations, it can be useful to run a web server on the phone and connect to that instead of to the notebook server. I find the lightweight web server lighttpd attractive for this. I install and run it as follows:

    # install lighttpd
    apt-get install lighttpd
    # configure it suitably (this would probably require a separate tutorial)
    nano /etc/lighttpd/lighttpd.conf
    # run the web server only when required
    /etc/init.d/lighttpd stop
    update-rc.d -f lighttpd remove
    lighttpd -D -f /etc/lighttpd/lighttpd.conf

At some point, I might run an ssh server or an ftp server on the phone as well, though I have not felt the need for that yet.

Conclusion

A phone with spare storage space, reasonably powerful processor and decent RAM is a more powerful computer than the mainframe computers of my student days. If I install the right software, it is perfectly capable of running most of the programs that I might need while travelling, and I can avoid carrying my laptop all the time.

With all the programs that I have installed GNURoot Debian takes up about 4GB of space in the internal storage. Since (a) GNURoot Debian runs directly on the Android kernel and (b) I do not use any graphic environment in Linux, the RAM usage is quite low (it typically uses less RAM than the GMail or Dropbox Apps and only a tiny fraction of what the Chrome browser uses). Compared to my Intel i7 powered laptop with 8GB of RAM, the phone is quite slow in running computationally intensive tasks. But I have found the setup quite usable on budget phones like the Moto G4 Plus and the Moto G5 Plus.

Note for Geeks

This note is for those who are wondering how all this works without rooting the phone though normally only root can install software. Yes, it is a lot easier to install a Linux distribution in a rooted phone, but GNURoot Debian is able to pull this off on a non rooted phone. It uses a tool called PRoot that allow non root users to run almost any Linux commands in a jailed shell environment. Inside this jail, you appear to have all the capabilities of root. For example, you seem to be able to install new programs and create files in any folder. PRoot makes you think that you are writing to /usr/bin, but you are actually writing to something like /home/me/jailed_root/usr/bin. Technically, PRoot is a user space implementation of Chroot; it uses the Ptrace system call that allows one program (for example, a debugger) to run another program under its control, intercepting all its system calls. When you try to write to /usr/bin, PRoot (via Ptrace) intercepts that call and converts it into a write to a folder inside the jailed environment, say /home/me/jailed_root/usr/bin.

Why Aadhaar transaction authentication is like signing a blank paper

Using Aadhaar (India’s biometric authentication system) to verify a person’s identity is relatively secure, but using it to authenticate a transaction is extremely problematic. Every other form of authentication is bound to a specific transaction: I sign a document, I put my thumb impression to a document, I digitally sign a document (or message as the cryptographers prefer to call it). In Aadhaar, I put my thumb (or other finger) on a finger print reading device, and not on the document that I am authenticating. How can anybody establish what I intended to authenticate, and what the service provider intended me to authenticate? Aadhaar authentication ignores the fundamental tenet of authentication that a transaction authentication must be inseparably bound to the document or transaction that it is authenticating. Therefore using Aadhaar to authenticate a transaction is like signing a blank sheet of paper on which the other party can write whatever it wants.

All this was brought home to me when I bought a new SIM card recently and was asked to authenticate myself with a finger print. The employee of the telecom company told me that there was a problem and I needed to try again. Being a little suspicious, I stretched forward and twisted my neck to peep at the computer screen in front of the employee (this screen would otherwise not have been visible to me). My suspicion was allayed on seeing an error message on the screen and I tried again only to get the same error message. After three attempts, the employee suggested that I come again the next day. Back home, I saw three emails from UIDAI (Unique Identification Authority of India) stating “Your Aadhaar number ___ was used successfully to carry out e-KYC Authentication using ‘Fingerprint’ on ___ at ___ Hrs at a device deployed by ___.” Note the word successfully.

That is when I realized that the error message that I saw on the employee’s screen was not coming from the Aadhaar system, but from the telecom company’s software. That is a huge problem. This conclusion was corroborated the next day when after one more error message, I found that the employee had left one field in the form partially filled and the error message disappeared when that was corrected.

Let us think about why this is a HUGE problem. Very few people would bother to go through the bodily contortion required to read a screen whose back is turned towards them. An unscrupulous employee could simply get me to authenticate the finger print once again though there was no error and use the second authentication to allot a second SIM card in my name. He could then give me the first SIM card and had over the second SIM to a terrorist. When that terrorist is finally caught, the SIM that he was using would be traced back to me and my life would be utterly and completely ruined.

Actually, even my precaution of trying to read the employee’s screen is completely pointless. The screen is not an inseparable part of the finger print reader. On the contrary. the fingerprint reader is attached by a flimsy cable to a computer (which is out of view) and the screen is purportedly attached to the same computer. It is very easy to attach the fingerprint reader to one computer (from which a malicious transaction is carried out) and attach the screen on the counter to another computer which displays the information that I expect to see.

Another way of looking at the same thing is that a rogue employee of the telecom company could effortlessly execute what is known in computer security as an MitM (Man in the Middle) attack on the communication between me and the Aadhaar system. In fact, I see some analogies between the problem that I am discussing and the MitM attack described by Nethanel Gelerntor, Senia Kalma, Bar Magnezi, and Hen Porcilan in their recent paper (h/t Bruce Schneier). Neither I nor the Aadhaar system has any way of detecting or foiling this MitM attack.

I think the whole model is fundamentally broken, and Aadhaar should be used only to verify identities, and not to authenticate transactions. Transaction authentication must happen with a thumb impression, a physical signature, a digital signature or something similar that is inseparably bound to a document.

The menace of link rot

I have been writing a blog on financial markets for nearly a dozen years now, and it is distressing to find that when I look up one of my older posts on that blog, many of the links in that post no longer work. This phenomenon is known as link rot.

Recently, there was a reorganization of our Institute web site and we had to set up a fair amount of URL redirection and rewriting to keep the links working. To check that everything was fine on my webpage, I ran LinkChecker to check that all internal links are working. Then out of curiosity, I ran a check on external links as well on my blog year by year:

  linkchecker --check-extern -r 1 --no-warnings --no-status URLyyyy

where URLyyyy is the url that displays all blog posts for year yyyy. What I found was that there were no broken links in the blog posts of 2016 and only one broken link in 2015. Beyond that, there were 10-20 broken links in most years (as many as 25-35 broken links in some years) out of about 300-500 links each year. This means that about 5% of all links break after about 2 years.

I could not see any pattern in the broken links. Links to government reports and court judgements from the official websites were broken presumably because they reorganized their websites and changed the URLs. Working papers and other academic papers from academic websites were broken too which makes me grateful to SSRN and ArXiv for providing permanent URLs for academic papers. Some links to media sites were broken because they went behind a paywall. I found many large financial institutions and exchanges in my list of broken links. It makes me wonder whether URL redirection and rewriting is so hard in practice for large organizations that spend millions of dollars on their websites.

What can we do about this? One solution is the Internet Archive Wayback Machine. I was happy to find that I could access some of my broken links using the Wayback Machine; at least, it confirmed that the link was correct at the time of the blog entry. Another solution that I have adopted (not as systematically as I would like) is to convert the web pages to PDF and save them locally when I link to them. This way, at least I would not lose access to these pages when link rot sets in. Unfortunately, that does not help my readers.

Predicting human behaviour is legal, predicting machines is not?

I read this Wired story about some hackers being sent to jail for “hacking” slot machines in US casinos. “Hacking” is probably the wrong word to use for this: they made money by predicting what the slot machine would do by observing it carefully, and using their knowledge of the insecure random number generator used in the software of the slot machines. It appears therefore that it is illegal to predict what a machine would do by figuring out its vulnerabilities and observing its behaviour.

The irony of the matter is that the entire business model of the casinos is built on figuring out the vulnerabilities of the human customers, predicting how they would bet under different situations and designing every minute detail of the casino to exploit these vulnerabilities. The New Yorker had a story five years ago about how a casino was redesigned completely when the customer profile changed from predominantly older male customers to more women:

So Thomas redesigned the room. He created a wall of windows to flood the slot machines with natural light. He threw out the old furniture, replacing it with a palette that he called “garden conservatory” … There are Italian marbles … Bowls of floating orchids are set on tables; stone mosaics frame the walkway; the ceiling is a quilt of gold mirrors. Thomas even bought a collection of antique lotus-flower sculptures

Casinos “monitor the earnings of the gaming machines and tables. If a space isn’t bringing in the expected revenue, then Thomas is often put to work.” The design is optimized using a massive amount of research which can justifiably be called “hacking” the human brain. If you look at the Google Scholar search results for the papers of just one top academic (Karen Finlay) in the field of casino design, you will see that she has studied every conceivable design element to determine what can cause people to bet more:

  • A comparison of ambient casino sound and music: Effects on dissociation and on perceptions of elapsed time while playing slot machines
  • Casino decor effects on gambling emotions and intentions
  • Assessing the contribution of gambling venue design elements to problem gambling behaviour
  • The Influence of Casino Architecture and Structure on Problem Gambling Behaviour
  • Measuring the Effects of Pictorial and Text Messages on Memory and Gambling Intentions Within a Casino Environment
  • The Effect of Visual Stimuli in Casinos on Emotional Responses and Problem Gambling Behavior
  • The Effect of Match and Mismatch Between Trait and State Emotion on At-Risk Gambling
  • Effects of slot machine characteristics on problem gambling behaviour

The more recent studies on human behaviour are done using a panoscope which:

features networked immersive displays where individuals are absorbed in an environment (12 feet in diameter) that surrounds them on a 360-degree basis. … Use of these panels creates a totally immersive life-like experience and facilitates the delivery of these manipulations. (Finlay-Gough, Karen, et al. “The Influence of Casino Architecture and Structure on Problem Gambling Behaviour: An Examination Using Virtual Reality Technology.” ECRM2015-Proceedings of the 14th European Conference on Research Methods 2015: ECRM 2015. Academic Conferences Limited, 2015.)

I do not see how this kind of attempt to fathom the workings of the human mind is much different from the hackers buying scrapped slot machines and figuring out how they work.

The better way to think about what is going on is to view it as a bad case of regulatory capture. The Wired story says that “Government regulators, such as the Missouri Gaming Commission, vet the integrity of each algorithm before casinos can deploy it.” The sensible thing to do is for the regulators to decertify these algorithms because the random number generators are not secure and force the casinos to use cryptographically secure random number generators. The casinos do not want to spend the money to change these slot machines and the captured regulators let them run these machines, while taxpayer money is expended chasing the hackers.

Perhaps, we should be less worried about what the hackers have done than about what the casinos are doing. Unlike the vulnerabilities in the slot machines, the vulnerabilities in the human brain cannot be fixed by a software update. Yet hacking the human brain is apparently completely legal, and it is not only the casinos which are doing this. Probably half of the finance industry is based on the same principles.

Setting up a Raspberry Pi as a home file server

Since tastes and needs differ widely within our family, the laptops and devices at my home run Linux, Windows 10, Android and iOS. Sharing data between devices running diverse operating systems is a huge pain. We often end up using email, Dropbox or Google Drive, though it is utterly silly to move a file half way round the world to share it with somebody within arm’s reach.

This is where the Raspberry Pi comes in. It is cheap enough to be within the budget of almost anybody who runs a WiFi network at home, and it is perfectly capable of running a file server accessible from Windows, Linux, Android and iOS. This post describes how I set it up for this purpose using Arch Linux. The Raspberry Pi (RPi for short) can run many different operating systems, but I chose Arch Linux ARM because it is quite powerful and also because I run Arch Linux on my laptop.

Prerequisites

  • RPi starter kit. At a minimum, we need:
    • Raspberry Pi 3 (If you use a RPi2, you will need a USB WiFi adapter)
    • Micro USB Power Supply
    • 16 GB SD card (8 GB might be adequate) and
    • LAN cable.

      Neither a keyboard nor a display are needed as the RPi will run headless.

  • An external USB hard disk of adequate capacity (say 1TB or more) with an independent power supply. Alternatively, a powered USB hub can be used to connect a USB hard disk without independent power supply.
  • A WiFi router (administrative rights are needed)
  • A computer running Linux to setup the SD card
  • Optionally, a music system and an unused mobile phone to create a networked music system.

Continue Reading

The blockchain as an ERP for a whole industry

In the eight years since Satoshi Nakomoto created Bitcoin, there has been a lot of interest in applying the underlying technology, the blockchain, to other problems in finance. The blockchain or the Distributed Ledger Technology (DLT) as it is often called brings benefits like Byzantine fault tolerance, disintermediation of trusted third parties and resilience to cyber threats.

Gradually, however, the technology has moved from the geeks to the suits. In the crypto-currency world itself, this evolution is evident: Bitcoin was and is highly geek heavy; Etherium is an (unstable?) balance of geeks and suits; Ripple is quite suit heavy. History suggests that the suits will ultimately succeed in repurposing any technology to serve establishment needs however anarchist its its original goals might have been. One establishment need that the blockchain can serve very well is the growing need for an industry-wide ERP.

ERP (enterprise resource planning) software tries to integrate the management of all major business processes in an enterprise. At its core is a common database that provides a single version of the truth in real time throughout the organization cutting across departmental boundaries. The ERP system uses a DBMS (database management system) to manage this single version of the truth. The blockchain is very similar: it is a real time common database that provides a single version of the truth to all participants in an industry cutting across organizational boundaries.

To understand why and how the blockchain may gain adoption, it is therefore useful to understand why many large organizations end up adopting an ERP system despite its high cost and complexity. The ERP typically replaces a bunch of much cheaper department level software, and my guess is that an ERP deployment would struggle to meet a ROI (return on investment) criterion because of its huge investment of effort, money and top management time. The logical question is why not harmonize the pre-existing pieces of software instead? For example, if marketing is using an invoicing software and accounting needs this data to account for the sales, all that is really needed is for the accounting software to accept data from the marketing software and use it. The reason this solution does not work boils down to organizational politics. In the first place, the accounting and marketing departments do not typically trust each other. Second, marketing would insist on providing the data in their preferred format and argue that accounting can surely read this and convert it into their internal format. Accounting would of course argue that marketing should instead give the data in the accountant’s preferred format which is so obviously superior. Faced with the task of arbitrating between them, the natural response of top management is to adopt a “plague on both houses” solution and ask both departments to scrap their existing software and adopt a new ERP system.

It is easy to see this dynamic playing out with the blockchain as well. There is a need for a single version of the truth across all organizations involved in many complex processes. Clearly, organizations do not trust each other and no organization would like to accept the formats, standards and processes of another organization. It is a lot easier for everybody to adopt a neutral solution like the blockchain.

A key insight from this analysis is that for widespread adoption of blockchain to happen, it is not at all necessary that the blockchain be cheaper, faster or more efficient. It will not be subjected to an ROI test, but will be justified on strategic grounds like resilience to cyber threats and Byzantine actors.

The only thing that worries me is that the suits are now increasingly in charge, and cryptography is genuinely hard. As Arnold Kling says: “Suits with low geek quotients are dangerous”.

In the sister blog during September-December 2016

The following posts appeared on the sister blog (on Financial Markets and their Regulation) during September-December 2016.