Let My EPUBs Go

Lately, I have been finding myself getting increasingly frustrated with the restrictions that are being placed on the content I choose to consume. In the so-called war on piracy, user experience is being sacrificed and I have no choice but to compromise or find workarounds to enjoy my content the way I want to. I will offer e-books as an example here. Music, movies and TV are still problematic, but thanks to streaming services (Spotify, iTunes, Play Music, Netflix, HBO Go) these are not as bad as e-books.

So channeling my inner Lewis Black, here goes:

E-books are easy to carry around to read - I already carry my phone and I can access thousands of them in my cloud library which takes up a tiny amount of physical space in some server farm somewhere. E-books can be synced across devices, so that I don't need to bother with bookmarks to hold my position. This is a god-send when I am feeling especially lazy and will just pick up the nearest device to read - my phone, my tablet or my laptop. That's it! These are the only two things I care about in a reading setup - ease of access and cross-device sync. If the setup has a well-designed display and smooth transitions between pages, that's an added bonus.

I am a member of the Singapore National Library Board, and in addition to the eight items I can check out at a time, I am also entitled to six e-resources. NLB, like tens of thousands of other libraries globally, uses Overdrive, which is the largest collection of e-resources in the world and unfortunately, provides a very poor experience considering my two concerns above.

First, ease of access. I have used Google Play Books for a couple of years now, and I am very happy with the experience. Books are a click away to read on all my devices and I can choose which books to keep locally on my device. Everything works from my Google ID, with virtually no hiccups. 

On the other hand, Overdrive tends to be unnecessarily problematic. First, I need to create an Overdrive ID. Then, I need to add my library into my Overdrive profile and log in using my library ID. On every device, I am logged out of the library system after a certain amount of time, so browsing the catalog requires me to log in again. But wait. Overdrive allows me to use Adobe Digital Editions (ADE: the go-to DRM solution for almost every major e-book publisher) to authorize the books on my computer to read offline. Unfortunately, I use Linux and there isn't any version of ADE that I can use. The only version that I can run under Wine is 1.7, which compared to the current 4.0 version, is positively prehistoric. Okay, so I get ADE installed (creating an Adobe ID) and then try to download a book to my computer that I checked out from the library on my phone.

Computer says no. Apparently because I am authorized in my Overdrive phone app with the Overdrive ID and ADE on my computer with the Adobe ID, the book cannot be read in both places. Oh no no, Overdrive helpfully allows me to authorize ADE with my Overdrive ID, but

Note: These instructions are for ADE 2.0 or newer. We recommend always using the latest version of ADE. You can learn how to install Adobe Digital Editions on your Windows or Mac computer here.

I would if I could! Remember how I cannot use anything beyond ADE 1.7 on Linux? This option goes out of the window. OK, so how about I sign in to the Overdrive app with my Adobe ID? No go.

Note: You may also authorize newer versions (3.2+) of the OverDrive app with an Adobe ID, but you don't need to. When you first launch the OverDrive app, you'll be required to log in with an OverDrive account, which authorizes the app automatically.

In other words, I will be forced to create an Overdrive ID. Worse, because the initial authorization is automatic, I have to deauthorize the app first, then reauthorize with the Adobe ID. Does it work? 

Oh no, I borrowed the book initially when I was authorized with Overdrive, so changing to Adobe voids the authorization, instead of actually reauthorizing it with the new ID like any sane piece of software would do.

Three IDs and I am still nowhere close to reading the book that I want to. OK, so how about I return the book and check it out again? (a.k.a. have you tried turning it off and on again?) Hopefully, this should result in a single authorization and I can finally get access where I want. But! I cannot return a book I checked out with a different authorization if I cannot download it. Why would Overdrive do that? If I am logged into my Overdrive account and my library account, I should be free to check out and return books. I can almost understand DRM restricting what I read, but being locked out of the system for all book transactions is crazy. At this point, my only option is to wait for the book to expire and be automatically returned. 

Of course, if you are familiar with Overdrive, you must be asking why I don't read directly in my browser? After all, that's the only option that Google Play Books offers, so it's a direct equivalent. Well, there are two minor reasons: First, it's a matter of principle. If Overdrive offers a feature, I expect it to work. If I have the option for reading offline via ADE, then I want to be able to use what is advertised. Second, Google Play Books does allow you to get your files offline, either via Takeout for uploaded files or directly for purchased books.

However, the main reason for wanting a different solution for reading e-books is the abysmal cross-device syncing in Overdrive. I have had about a 10% success rate getting a book to correctly sync. This was before the ADE fiasco, so I was moving between phone, tablet and browser.

First, if I check out a book from the library using my Overdrive account, it is not automatically available in my Overdrive account on all my other devices. I am required to log in to the library account to be able to see what books I have checked out. Contrast this with Google Play Books, which helpfully sends a notification to all my devices when I upload or purchase a new book. Tap on the notification and I can start reading immediately.

Second, there seems to be some major bugs in the syncing algorithm. For some reason, it is unable to identify what was the last device I read on. In a few cases, if I started reading on my phone with my last session also on the phone, it offered to put me back to where I was at the start of my last session! At other times, it managed to sync the chapter correctly, but could not get me to the correct page in the chapter. I have never had such an issue with Google Play Books, and the lack of this ability has forced me to look at other alternatives.

Unfortunately, as the publishing industry is mired in such a deep love affair with DRM, there is no easy, viable alternative. I can use a pipeline like Wine+ADE+Calibre+DeDRM plugin, but that goes against the usage rights of the e-book, and I don't want to do that because it will provide short-term justification to the publishers to further clamp down on e-books and charge readers and libraries even more for them.

If you think about it, the concept of e-books being handled as paper books is all wrong. The marginal cost of another e-book is minuscule compared to that of another paper book. Within this premise, the concepts of hold lists, limited copies, delayed online publishing windows can only be justified by publishers trying to squeeze as much money as they can out of this new medium. In fact, according to this article from GigaOm,

...“friction” may decline in the ebook lending transaction as compared to lending print books. From the publisher viewpoint, this friction provides some measure of security. Borrowing a print book from a library involves a nontrivial amount of personal work that often involves two trips-one to pick up the book and one to return it. The online availability of e-books alters this friction calculation, and publishers are concerned that the ready download-ability of library ebooks could have an adverse effect on sales.

In other words, making e-books difficult to access, especially from libraries, is in the interest of the publishing industry. It will probably be some time before there is any meaningful change that comes about; until then I'm going to try to stick to paper books from NLB. 

RIP, Mr. Lee Kuan Yew

I finished reading the fantastic book Trespassing on Einstein's Lawn by Amanda Gefter yesterday. In the book, she presents the arguments that lead to the conclusion that in physics, there is nothing that is invariant in all reference frames. In other words, there is no ultimate reality; all reality is observer-dependent. This point of view particularly resonates with me because I believe that every person has a perspective on things that to him or her seems correct. Of course, in our cultural and social norms, we do have a pseudo-ultimate reality enshrined in our traditions, laws and etiquette, but these are man-made constraints and not inviolable otherwise.

What is the point of this? The point is that while many eulogies and obituaries will be written for Mr. Lee Kuan Yew, Singapore's first prime minister who passed away this morning, my perspective of living in Singapore while not being born and brought up here helps me understand the enormity of what he managed to accomplish. My point of view of what Singapore is today is influenced by the fact that having grown up in Mumbai, I can see the different directions the story could have gone if not for this man of iron will.

Let's start with the similarities. Mumbai and Singapore are both islands of roughly the same area (600 vs 700 sq. km.). Both were ruled by the British before independence, both have the advantage of being natural ports that helped foster initial settlements and trade. Both attained independence within two decades of each other. Emerging from the convoluted process of gaining independence, both were tried by communal tensions (Mumbai indirectly due to Partition, Singapore between the Malay and Chinese communities).

Of course, this isn't to say that Mumbai and Singapore are identical. Mumbai is a part of a much larger country and policies that influence the nation may not have a directly beneficial effect on the city. Combine this with the tri-level governance of nation, state and city and the sheer force of population makes it a wondrous story in its own right.

On the other hand, the lack of being part of a larger country was also a big handicap for Singapore. With no natural resources, no agriculture, no oil, no minerals, no drinking water, there was virtually no hope for this tiny city-state to survive. The only three things the new nation possessed were its port, its existing British infrastructure in the downtown area and its people - a largely uneducated and illiterate populace of immigrants.

As a commenter on Reddit said, given these initial conditions, I would have restarted my game of Civ4. Enter one of the greatest gamers ever seen - Lee Kuan Yew. After crying on television about Singapore being thrown out of the Malaysian Federation, he set about the task of building Singapore up. He was a visionary like no other.

While most emerging nations are hesitant about foreign investment fearing foreign influence, he encouraged it because he saw that Singapore had nothing it could export or manufacture on its own. Backed into a corner, he encouraged foreign companies to set up base in Singapore, offering them tax incentives and ease of doing business. The experience gained by the population from working in these companies helped local banks and technology companies climb to be shoulder-to-shoulder with some of the best in the world. Think about that: the man saw that the only option was to take which most other countries in its place feared, and he took it and played it so that it benefited the country.

Faced with a severe housing shortage in the 50s and 60s, he set up the Housing Development Board (HDB) to make sure that most people could own their houses and feel they own a part of the nation. It is thanks to his vision that fifty years later, Singapore ranks as one of the highest in the world in terms of home ownership rate (>90%). While the technicality of the term "home ownership" may be debated - all HDB flats are technically on a 99-year lease from the Board - it is nevertheless a long-term security that citizens possess. Contrast this with MHADA, where the mere name conjures up pictures of run-down, cramped buildings.

Faced with a lack of drinking water and importing water from Malaysia, he invested in creating water catchment areas on the island, recycling water as well as desalination technologies (all of which were years away from being commercially viable) as a result of which Singapore aims to be completely self-sufficient water-wise by 2061. Think about that: An island with barely any drinking water and limited land aims to have enough water to support a population of millions. Facing a local populace not tied together by language, he made education in English compulsory (along with a mother tongue language) as that is the language of science, technology and business worldwide. Contrast that with the language politics that we see in India with English and Western culture looked upon as destroying all that is good and holy. To attract top talent to government and keep it corruption-free, he pegged cabinet members' salaries to that of top-earning executives in private companies. Look at the beauty of that idea: if you can earn that much, there is less need for you to be corrupt when you wield power and earning that much can be an incentive for you to get into government, not just the need to do good for the people.

I could go on with other examples of vision and foresight that LKY had, but there was also the other less positive side of things. He ruled with an iron fist, bankrupting opponents (thereby making them ineligible to run for office) via costly lawsuits, controlling the media and imprisoning whoever was considered to be an internal threat to Singapore. We still see the lingering effects of these even though some of the policies have loosened thanks to the democratization of speech via the Internet. Nevertheless, the way I look at it, his view was that the ends justify the means. We try something and if it works, fine; if not, throw it away, try something else.

Yes, in terms of conventionally defined Western freedoms, Singapore lags far behind. This however begs the question: are such freedoms good for their own sake? I will leave the philosophers and the policy-pundits to debate this, but I believe that from the survivalist tendencies that Singapore emerged, this was the best way forward and we are gradually seeing a change in the idea of freedoms here. Freedom of expression and speech engender creativity and the incipient arts and culture scene and the increasingly vocal political space are examples of areas where boundaries are being pushed and modified. It will be a test of the current generation of leaders if they can maintain the trajectory that Singapore was launched on, while still adapting to the times.

As more or less an internal observer with an external background, I see LKY's policies' influence wherever I go. Whether it is taking a flight out of Changi (he pushed for an airport at the edge of the island, when multiple studies advocated the expansion of an existing inland airport like Sahar), taking the MRT to work (the largest public works project of its time, pushed by LKY because he believed that land was too precious for an all-bus system), heading into the research cluster of Singapore (born out of the Industrial Research Unit under the Economic Development Board set up by him), having lunch at a food court or hawker centre (LKY's government organized the food services sectors by setting up markets and formal hawker centres in housing and commercial estates) or going for a walk along the park connector (LKY pushed for greening Singapore to such a degree, that the motto for the National Parks Board today is not Garden City, but City in a Garden), I see the gigantic influence he has had on the place I live. More importantly, I am also able to see how easily things could have gone south. Giving in to special interests, pandering to communities on the basis of race or religion, copying principles from other countries without pausing to consider local applicability or rejecting viable ideas because they didn't emerge internally would all have sabotaged one or more of these.

And that prescient understanding of Singapore and its position in the world along with an iron will while being open to trying new things is what made him such an effective leader. Singapore is truly poorer for having lost its greatest caretaker and his loss will resonate for years as people stop to ask, "What would LKY have done?"

RIP, Lee Kuan Yew.

Headspace

I got my hands on a Bluetooth headset (the very comfortable LG HBS-750) a few weeks ago for my commute. I always prefer wireless headsets in the train as there are no wires to be pulled or tangled in the crush. Another reason I enjoy my personal audio in the train is that it gives me much needed headspace, especially when decompressing from work.

I have started listening to some podcasts so I'll chronicle them below. Any recommendations would definitely be welcome. As an aside, I use Player FM as my podcast app, and it is brilliant with a well-designed UI, an extensive catalogue and good offline storage settings.

So, here are some of my favorite podcasts currently:

1. Serial: Well, of course! How can I not be listening to Serial given the hype surrounding it? For those who don't know, it is a long-form weekly podcast spanning almost three months that investigates the alleged murder of Hae Min Lee by her ex-boyfriend Adnan Syed in 1999. The podcast takes us through mountains of unreliable data and dozens of twists and turns with no clear end in sight. It is probably what a lot of criminal investigations are like, and to witness the uncertainty and inconsistencies in the case presented by either side is very edifying.

2. TwiT: This was an unusual one for me. I didn't really get into it at the beginning, because it seemed too slow for what I considered to be a tech news program. After giving it some time however, I realized it is more of a group discussion and has tons of banter that really liven up a two-hour long show. It has become one of the must-hear podcasts for me, and I look forward to it every Tuesday (Singapore time).

3. The Bugle Podcast: From the titans of comedy John Oliver (of Last Week Tonight fame) and Andy Zaltzman (of Cricinfo fame), this self-described "audio newspaper for a visual world" is irreverent and holds no punches in taking everything and everyone to task. I am guaranteed a good chuckle during my commute, much to the disconcert of my co-passengers.

4. No Such Thing As A Fish: Another British podcast, this time from the writers of the comedy panel show QI, it covers interesting facts every week. Loads of comedy and you do get to learn some interesting stuff. For instance, did you know that after the moon landing, Buzz Aldrin worked as a car salesman in a Cadillac dealership and didn't sell a single car in six months? This, of course, led the panel to question whether it was a good decision to have the dealership on the moon, but that's a different story. The name of the podcast comes from the fact that the common ancestor of what we call fish is also the common ancestor of all four-legged, land vertebrates and therefore, there is really no such thing as a fish.

So, that's all from me for now. Do you have any podcasts that you'd recommend for me to add on? I'm still on the lookout for a daily news program that is available between 4AM and 7AM SGT, so that I can get some current news on my way to work. Something like BBC Asia would work well but I can't seem to find a good, relevant podcast. All suggestions are welcome!

A Picture is Worth A Thousand Words (Literally)

I have a healthy skepticism of the part of the job interview process where applicants are asked to write a super-efficient piece of code to create what amounts to a cog in a much larger machine. In my experience as a researcher and developer, most of my time is spent finding the cogs bringing the machine to a grinding halt, also known as debugging, also known as staring at sections of code for minutes on end as your monitor goes to sleep.

One of my favorite methods of debugging is to visualize what I am dealing with wherever possible. Indeed, one of the reasons I got into the field of image processing and computer vision was because generally, when you don't know what's going on, you can look at an image output and get a pretty good idea of what's happening.

Recently, I have been looking into evaluating a classification technique that I have been working with on a text categorization problem. I am familiar with the fundamentals of text classification, as visual classification has long been inspired by the former, but have never worked in the domain before. In order to make what follows a bit easier for non-CS people to follow, I'll try to explain the classification problem in simpler, qualitative terms. Feel free to skim through it if you're comfortable with the field.

Who's who?

So here I am looking at a collection of about 5,000 webpages crawled from the websites of multiple universities. The collection is split into a training set and a testing set. As the names imply, the former is used for teaching our classification algorithm what it is to recognize and the latter is used for testing its performance. The actual categorization problem is a bit more complex, but for the purpose of explanation I'll say we want to distinguish between webpages belonging to faculty (the positive class) from the webpages belonging to non-faculty (the negative class, comprising of students, projects, clubs, etc.). Each document is labelled with its class, and the labels are supplied to the classifier for training, but only used for comparing its outputs in testing.

How do we go about this? We need to structure each document into a form from which its meaning can be easily extracted. One of the most popular techniques from years of research in text analysis is to simply take all the words appearing in a document and count their occurrence. The order of the words is ignored, and the output we get is basically a count of how many times a word in an arbitrary vocabulary or dictionary appears in the document. This is termed as a document vector. There may be additional preprocessing and normalization steps involved, but I'll skip those for now. As an example, a document like:

apple banana apple papaya

will result in a vector:

apple=2 banana=1 orange=0 papaya=1 watermelon=0 ...

for a fruit-based dictionary. Each vector thus supplies the classifier with information about which words occur in the document and how frequently. The classifier then looks for common words in documents belonging to the same class, and uses them to try to assign new, unseen documents to the correct classes.

So what's this got to do with visualization?

Now, the problem I was running into was that my algorithm kept giving me excellent performance on the training set, but not on the testing set. This is very bad for any classifier because it means that it is not able to work on anything it hasn't seen before, making it useless. It is not an uncommon issue though. The classifier is ending up overfitting the data; it is placing too much emphasis on the data it has been given at the expense of leaving little wiggle room to deal with ambiguity in unseen data. This is like the anecdote of the kid who complains to his mother that he could not solve an addition problem in class because he had been taught to do it with apples and the problem asked him to add oranges. Similarly, what we want the classifier to realize is that it is certain words that matter, not all of them.

So, I go about trying to fix the apparent overfitting. This is done by something called regularization, which is a fancy word for penalizing your algorithm when it tries to improve its performance by reducing its flexibility too much. I tried different values for the regularization parameters, different regularization techniques, different parameters for other components of my algorithm. Nothing worked.

Puzzled but quite sure that something should have worked by now, I thought of checking the data. But 5,000 documents containing some 7,500 words overall are not easy to wade through. That's when my instincts kicked in, and I went to the comforting familiarity of images.

Visualization of the positive training set

I visualized the entire positive class of document vectors in the training set as the image above. The vertical dimension consists of the words in the vocabulary, the horizontal dimension consists of the documents, and the brightness of each pixel tells us how frequently a particular word occurs in the corresponding document. There is some normalization done as well, which is responsible for the relatively uniform brightness you see in the horizontal dimension.

Notice there are some bright rows in the picture. I looked back into the vocabulary for the words these rows correspond to and these turned up being words such as "professor", "faculty", "department", "research", etc. - dead giveaways for the positive class, especially when more than one of them occurs on the webpage.

Now, let's take a look at the testing set of the positive class.

Visualization of the positive testing set

Notice the lack of bright horizontal lines or any significant structure for that matter. This immediately violates one of the preconditions for successful classification - the distribution of words in the training and testing sets should be approximately the same (the other condition is that there should be some distinction between the positive and negative classes). No classifier can do well on such a problem, because there is no information about the testing set that can be learned from the training set.

That long-winded explanation is how I realized that there was something wrong with the data. I checked and yup, there was some corrupt data that had caused the document vectors to be improperly calculated. Once that was fixed, the correlation between training and testing performance immediately manifested itself.

Moral of the story? Don't trust the data, especially if you're in unfamiliar territory.

Greetings, planet!

What? Did you really think I wasn't going to open with some variation on "Hello, world!"?

Anyway, I just thought of starting a blog, because I certainly don't tweet or share as much as I used to. Ironic, considering that I am working on social media analysis these days. Moreover, I read this wonderful article on Lifehacker the other day, which says that no matter how little you think you know about something, you are still an expert to someone. Therefore, I decided to start getting some of my thoughts out there, and hopefully generate some new ideas.

I really don't know what this blog is going to be about. Maybe some personal stuff, maybe some Formula 1, maybe some computer vision stuff. Of course, without any idea of what I want this to be, but not wanting to add another blog to the catch-all of "Random Thoughts", I decided to go a bit technical with the title.

I'll try to post regularly and see how I take to blogging, so...