How good is our latest Singapore Library APIs? An Honest Review

Photo by Thomas Kelley on Unsplash

Recently, the National Library Board (NLB) folks shifted their publicly available API from the older SOAP API to the more modern REST API (Interested public can apply to this API here). Although I had to spend time migrating my code to the new REST API, I felt this was a much needed refresh to keep the service up to date with modern tech stacks. Those interested can about my side project using the NLB API (here, here, here and here). I feel this revamp also signals some level of commitment from NLB to improve and maintain their API service, and I am grateful for their thankless dedication to this.

With that in mind, I want to share my learnings from this migration process and provide some honest feedback as an end user of their API.

First impressions

Firstly, when the NLB folks emailed me about the API change with my updated API keys, I realised there was no changelog provided. A changelog would help users identify any API changes that may break their existing integrations of the API service. From my past experiences using APIs from tech companies and government agencies, when API changes are going to happen, users should expect a changelog. In fact, companies plan their API changes in advance and communicate when these changes will occur, so that their API users can plan their own refactor cycles accordingly.

Maybe the NLB team felt that since their users had to shift to an entire new API implementation, it makes sense for their users to just review the entire documentation and re-implement the API from the ground up. Maybe the NLB team believes that no one is actively using their APIs in production, so such a change wouldn’t affect any mission critical services.

The NLB team did email me on 14th July to say that they are depreciating the old SOAP API on 14th Sept, giving me a two month window to shift my API according. A two month window to refactor my code is definitely better than having the service suddenly changed or cut off. Regardless, I had two months to figure out on my own what I had to do to migrate from the NLB’s SOAP to REST API.

The Endpoints

The NLB folks were really quick to remove their old SOAP documentation, so I can only show a Github repo of the old SOAP example implementation by Jiayu Yi (thanks Jiayu. Much of my earlier SOAP implementation was based on your code). The SOAP API has three endpoints that extracts information of physical books:

1. get_availability_info: This provides information on which libraries the book are residing, and if the books are available for borrowing.
2. get_title_details: This provides more detail book information, such as book titles, book descriptions, book classifications, etc
3. search_response: This allows users to search for books in our Singapore libraries based on certain search criteria

Fig 1 — NLB SOAP code example from Jiayu’s Github Repo

The REST API (Fig 2) has a new “GetNewTitles” endpoint that isn’t in the old SOAP API. While this did not directly affect any of my current API integrations, it does indicate that the REST API may have new features that differ from the old SOAP API. Not having that changelog makes me worry about changes that may seriously impact what I can do with the API.

Fig 2 — Endpoints of the new REST API

The Search

I will focus on the Search endpoint (I shall call it Old Search, versus New Search for the REST API’s Search implementation) as search should have the most differences between the two implementations.

Search Parameters

Old Search allows search on four parameters: (1) Keywords (2) Author (3) Subject and (4) Title, while New Search allows search on (1) BRN (2) ISBN (3) Author (4) Subject and (5) Title. I never tested the Keyword search on the Old Search, so its removal doesn’t affect me. The newly added BRN and ISBN search parameters on the new Search are on book ID numbers that the NLB and international book community use to identify books, and I could potentially have a feature where a user searches for books by their BRN / ISBN.

(Naive) Search Speed Test

I did a quick check on both APIs’ performance, and the New Search is much slower.

Fig 3 — The first function runs the old SOAP API, while second function runs the new REST API.

On a side note, the Old Search seems to have some caching that makes repeated searches of the same term faster. This feature doesn’t seem to exist for the New Search. This will affect my web app’s performance as I am directly using this API to perform Search on the NLB books. While I will try to see how I can improve the preceived Search performance on my web app (maybe caching or some other method), I hope the NLB team will take note of this and consider improving their API performance.

Search Results and Quality

Huge disclaimer : Search quality is a huge topic. Google built an entire company out of providing good search, and my analysis on the search result is based on my experience as their end user who is thinking about what search experience I want my web app users to have.

Search Results

Firstly, using three different Title Search parameters (“Python”, “Python Programming” and “Singapore Cooking”), I found the Old and New Search APIs return a different number of records. The New Search’s BRN and ISBN parameters gave me no results, which is as expected.

Fig 4 — Search results on “Python”

Fig 5 — Search results on “Python Programming”

Fig 6 — Search results on “Singapore Cooking”

Search Quality — Removal of eBooks

One positive thing I realised was that New Search now didn’t have any ebook results. It was actually puzzling to me that the Old Search had ebook results, because both the Old and New APIs have a different endpoint for extracting eResources from the NLB. When using the old SOAP implementation, I had to filter out ebook results programmatically, so this is definitely is an improvement to me.

Search Quality — Irrelevant results?

Next, given I was searching on “Title”, I kind of expected that “Python” should be in the titles of my book searches. Both Old and New search returned about 6% and 13% of books that did not have “Python” in their title. I am assuming now that the “Title” parameter doesn’t just search the book’s title, but also searches on some other information tagged to the book. I cannot say if this is a good or bad thing, but it is definitely something I need to consider when using it in my app.

Search Quality — Missing values

I also found that surprisingly, New Search had a record with a missing title. I did not face this in the Old Search, so unfortunately, this is a huge negative on the New Search’s search quality. A user on my app gave me a keyword that he searched that previously broke my app (“Lee Kuan Yew”) and one can visibly see the missing values from the search result for both the Old and New Search. I am not sure if this is a common situation with information about library books, but this definitely requires more work to handle missing values on my web app. Currently, I am just removing these results with missing values from my web app.

Other Annoyances

In addition, there were a few annoyances from the New APIs for me. Firstly, all returned json parameter keys are in small caps, which differs from the CamelCase format in the Old API. “BID” is now “brn” (with no prior knowledge and documentation on this, I could only ask my friend about this) and “TitleName” is now “title”. There are also differences in the json output structures that got me to re-write quite a bit of my code to fit the new json output. The same situation happens for the other endpoints that I am not covering in greater detail in this post. Fortunately, none of the API changes broke my web app in a way that I cannot refactor.

The new API documentation was also not too clear, and I had to ask the NLB folks on how to use their API. Depending on when I ask them, their replies can be pretty quick, or may take a few days at times. I do feel their response time is pretty reasonable given that they should have other daily tasks to handle.

The NLB team did not provide any code examples on how to use their API service, but to be fair, not too many companies do so (Those who do know that providing code examples make it easier for people to try their APIs). The code examples of the Old API were provided by volunteers, and the NLB team did explicitly state that they are not responsible for the workability of those codes.

Book Available Info

While refactoring my code to understand individual book availabilities, I realised that the SOAP API had “StatusDate” and “DueDate” of each book.

Fig 7 — I kept all the raw code to show that I did not remove or rename any column names accidentally

However, when I was using the REST API, not only was the JSON response structured differently (which is still quite understandable), there is only one “date” parameter. I did manage to figure out what information I needed to show, as I had prior knowledge about what these dates could be from the SOAP API. I reckon it will be quite difficult for someone to figure this out if they were trying to use this API for the first time.

Fig 8 — Looking to the right side of the REST API JSON output (converted to DataFrame), you can only see a “date” variable

Fig 9 — A deeper exploration, and I really only see one “date” parameter

The most puzzling experience

There was also a puzzling situation where a parameter seemed to appear out of nowhere after I enquired the NLB team about how pagination of the Search results can be done. After making the enquiry, where I was told to use ‘lastIrn’ for the pagination, ‘lastIrn’ suddenly appeared in the documentation.

Fig 10— The latest Search API responses, with the lastIrn and irn outputs

I did get pagination to work from the team’s reply (I needed it for the tests in this post), but the engagement feels strange. I initially thought I missed this part of the documentation, but after looking at my previous API responses that I saved in a separate Jupyter Notebook, I am confident that the “lastIrn” parameter was never the API previously until after my latest engagement with the NLB team.

Fig 11— Previous New Search API Json response. Notice this output does not have “lastIrn“ and “irn”

Maybe the NLB team were making changes to their API while I was asking the question, so the timing could just be coincidental. However, public APIs should have a proper change cadence that is announced to their users way beforehand, and follow a structured phase out process. This is so that their existing API users have time to rectify any changes that they need to do to continually use the API service. Sudden changes to the API can easily break services that are dependent on these APIs, and would ruin people’s confidence in using these API.

If the New API is still in Beta and changes are to be expected on the fly (which is understandable), the NLB team could at least mention this explicitly to your API end users.

I am truly grateful for the NLB team’s effort in providing the API, and I want the team to understand that my feedback is to help improve the quality of this API, so that it will encourage more people to use your API services. No right minded API developer will consider an API service that is unstable, buggy, and that is prone to irregular and unplanned changes. No one would dare to use such an API on a production environment for their own services.

Concluding Thoughts

This has been a quick review of the NLB API. I had to rush it out before the old API gets phased out by 14th September 2023, and here is a summary of the good and bads of the New API

1. [ Positive ] REST is a modern API framework that more people will be familiar with, and hopefully, encourage more adoption of the API
2. [ Positive ] New Search now only have results of physical books, which should be its intended use
3. [ Positive ] The NLB team is quite quick in replying my email enquires, some times even replying them within the day itself
4. [ Netural ] Some results from the “Title” parameter had titles without the searched keyword.
5. [ Neutral ] The NLB team contacted me two months before the SOAP API’s deprecation to allow me some time to perform my migration.
6. [ Neutral ] New docs, while limited, is still better than the old ones.
7. [ Negative ] Slower New API performance.
8. [ Negative ] Missing output values in both the New and Old API.
9. [ Negative ] All response output keys were changed, but no changelog was provided.
10. [ Negative ] In the BookAvailableInfo endpoint, the older SOAP API had to dates “StatusDate” and “DueDate” while the REST API only had “dates”. Both new and existing API users would be confused by what “dates” may mean here
11. [ Negative ] Unknown parameter “lastIrn” popped up out of nowhere.

I hope I don’t sound awfully critical of the NLB team here, because I cannot stress enough how grateful I am for this API (is this the 3rd time in this post that I am saying this?). If anyone from the NLB team is reading this, I am happy to share my learnings to you guys directly. I am contactable through my Linkedin .

Next Steps

With my learnings on the new NLB APIs, I want to see how I can improve the search quality of my SG library app. My current search uses the most basic results directly off the NLB API, so I am considering to add more search parameters like BRN and ISBN, and also maybe increasing the search results to show more than my current maximum fifty NLB found results (No web app currently has no pagination on search results).

I will also more actively promote my app to more users (thanks to existing registered users) to see if really anyone will benefit from it, once I have more confidence in my app (and the NLB API service). Those interested can consider registering and using my web app at this link. Those who don’t know can read about this post I wrote previously. Note that as I am still using a free-tier for my web server, my web app may be slow. I will definitely pay for a more stable web server if more people are interested in this.

Lastly, I am happy to connect on Linkedin, where I share my tech learnings and journey as a freelance data analytics consultant. I was a Social Science major (Economics) turned tech start-up data analyst turned freelancer, so I am interested in an array of topics, from tech, analytics, urban planning, art, sociology, economics, GIS (Geographical Information Systems), philosophy, and basically how to view our world when many of these different viewpoints collide into one another.

References and Links

1. Latest NLB REST API documentation
2. Previous NLB SOAP API Github volunteer repo
3. My Linkedin
4. My SG library app — The reason why I am using the NLB APIs
5. My past NLB writings — here, here, here and here