I created a Web App using the NLB Open API
Sharing the web app I created for myself using the NLB Open API!
A Recap
My past posts has me go deeper into my motivations for this API project: (1) Scraping Singapore Libraries (2) Iterations for my NLB scraper (Github code provided) and (3) Why I tried scraping prices of books I read — a non-technical sharing.
However, here is a summary of my motivations again. I am an avid library user, and I love their web app. However, when I want to look at the available books in my “Favourites” on the NLB app / website, I have to spend quite a bit of time clicking books by book.
Adding to the user experience friction is that the mobile app may not always be as stable, and not all my favourite books show up. Maybe NLB never expected someone will bookmark more than 60+ books into their app.
Personally, I like to bookmark the library books of my interest, and see what books are available at a particular library. Hence, what I want is when I am at a particular library, I want a single view of my NLB books that are available in this library.
Before the NLB Open API
Previously, I wrote a web app using Streamlit (frontend), Google Sheets (backend), and Python to scrape books from my NLB “Favourites” section. I wouldn’t go further into this as I had written this in my previous post. Finally, my book availabilities by their library locations!
My old set up before using the NLB Open API had a few caveats:
- I could only run my scraper on my laptop, meaning between the time I scraped the books and the time I am at the library, the books that I want may be borrowed by someone else.
- The entire scraper can take as long as 20 minutes to complete (I can have 60+ books in my Favourites)
- Some of my books in my Favourites don’t get scraped due to some page loading speeds, causing them to not surface on my web app.
- Only I can use this “service” right now, because I have to run it locally on my laptop. I want to see if I could deploy this service in the cloud (Streamlit) and share it to anyone who wants a similar view.
This article is about me refactoring my previous workflow to utilise the NLB Open API to get books availability, and hopefully, “launch” this as a service that anyone interested can use.
The latest version of my web app
While I had to understand how to utilise the NLB Open API, I will not go through that in detail here. For those interested, I wrote a separate Git repo explaining my explorations the NLB Open API.
On my new web app, I have 4 pages (1) Main Page (2) Step 1 Get Yr Bks (3) Step 2 Know Yr Bks (4) zAbout Me.
Main Page
This page provides instructions on how to use the web app.
Get Yr Bks
This is where you log in into your NLB account to download your favourite NLB books. The NLB Open API doesn’t have any user level data, so the only way I could figure to get users’ favourite books is to scrape them. This means I needed the user to provide their credentials (username and password) to allow my headless scraper to login into their NLB account to extract all of the user’s favourite books. Fortunately, this headless web scraper set up could be done on Streamlit
My workflow’s main improvement now is I can use the scraped books info to make API calls into the NLB Open API to find their availabilities across all Singapore libraries. The performance of the API is a slower than my expectations, taking as long as 5 minutes to complete 50+ books, which means that it feels like an eternality on the phone, but this is still better than having to run my script only on my laptop.
To ensure that I don’t store any user data (login credentials, books data) from anyone, I require my users to download the book availability data as a csv file, and upload it into the next page to be able to filter through the different libraries. This flow definitely feels more cumbersome when I am dogfooding the app myself, but given my time constraints, I felt this was a necessary evil.
Know Yr Bks
This is where you upload the downloaded csv file to show where your favourite books are available
I added a summary view of all the available books to quickly understand which library has most of the books that I want
Caveats & less than perfect points
I don’t feel that my NLB web service provides the best user experience.
Firstly, as mentioned, the favourite books have to be downloaded from page Step 1, and upload into page Step 2. Users may also find it hard to locate the downloaded csv file on their smartphones or laptops. In other cases, the download button disappeared after the scraping is done, and I had to re-run my web scraper.
Secondly, the NLB Open API is quite slow. Making API calls for just under 60 books can take as long as 5 minutes. On my smartphone, this did feel quite slow as well. In addition, I realise that because of how the API is structured, which is really book centric, I kind of realise why NLB is unable to show a library-centric books availability view. If they want to do what I want to do, they would have to re-engineer their API structure, which is interestingly written in SOAP. I suspect shifting to REST or GraphQL might allow them to show library-centric views, and this is definitely something I am keen to explore as another side project.
All these said, this was as much time as I can commit. Done is better than perfect. Most importantly, my web app has been working well for me for the past month. I am now curious if anyone else can find utility in my web app too.
Final words
If you managed to reach to the end of this post, I appreciate you taking your time to do so. And this is the link to my web app. If you have an NLB account, feel free to try my web app by following my instructions. Lastly, feel free to give any constructive feedback, and I promise to see if I can work on it.
