What I learned from my Full-Stack Engineering Side Project as an Analytics Consultant

Photo by Scott Blake on Unsplash

Previously, I wrote about my analytics side project that I felt got me into the analytics industry. Now, I want to talk about another software engineering side project that I did, where I developed a web app from scratch and deployed it online for anyone interested to use. I will cover my motivations, how I went about doing it, what I learned, and what I achieved so far, as this project is still ongoing.

Motivations

I started this side project after five years as a full-time data analyst in two Singapore tech companies, and one year of experience as an freelance analytics trainer and consultant. I had two reasons for starting this project.

Firstly, I wanted to improve my backend and frontend engineering skills through hands-on experience. A stretch goal would be to use this project to transit into a full-stack engineer from the skills I learned from it. But I felt that having direct experience dealing with data on backend servers and frontend applications will make me a more all-rounded analytics consultant when I work on analytics projects with other engineers. This was why I didn’t consider a no- / low-code platform for my web app, as implementing the actual engineering tasks was more important to me than just having the web app up.

Secondly, I wanted to solve a personal real pain point. I am an avid Singapore library user who borrows physical library books. Unfortunately, our Singapore library web and mobile apps only show book availability at a book level. So when I am at a library looking for the books I had saved to borrow physically, I have to annoyingly click on every single saved book to know if it is available at the library that I am at. I would prefer if I could click on a library filter, and have the web app show me all my saved books that are physically available for borrowing from that library. Fortunately, I realised our Singapore National Library Board ( NLB ) has an API that I can use to build a simple web app to show books availability by libraries. And knowing how slow changes may happen within government agencies, I took it upon myself to make the change that I want.

With these two rather compelling reasons, I rationalised my worst case scenario was spending a few months building a web app for no one else but my own personal use, while learning some full-stack engineering skills to aid my analytics consultant role. While the opportunity cost of “a few months” may feel high for an working adult professional, I felt the potential upside was worth a shot to see how far I can go with this project ( I wrote a past post on how I think and plan about events with uncertain payoffs here ).

Like my analytics side project post, I shall just gloss over the boring parts:

1. I went with Python FastAPI as my backend and Jinja2 templates to serve my frontend, because I am more familiar with Python than JavaScript.
2. As I had an Udemy Business account then ( from our NLB ), I took a course on using FastAPI to build web apps, and this really helped a lot.
3. I also supplement my FastAPI learnings from books from NLB.
4. I used MongoDB ( freemium ) for cloud database storage.
5. I used Render ( $7 USD / month ) to host my web app.
6. Lastly, I added some HTMX on my frontend for some user interactivity.

And now, the interesting stuff!

The working web app

After six months of work ( two months learning, four months implementation ), I successfully deployed my web app on Render. As of April 2024, the web app is still alive here. Through word-of-mouth, I got a single highly active user who really loves to read physical books like me, while I also continue to use the app myself. I had some initial users churn too.

Learnings as an analytics consultant

Being my own product team, I witness firsthand perspectives from many different roles:

1. I am not a proficient UX designer or full-stack engineer — My app is somewhat functional, but my design and functionalities can be much improved.
2. Operational versus analytics engineering — As an engineer, I only cared if my web app works. However, to do certain analytics, I needed to engineer things like daily snapshots or capture specific events into my database. I realised my demands as a data analyst adds more work to the engineer, and at times, the engineer me got a bit frustrated by the event tracking requirements from the analyst me.
3. Server side event tracking — When I started to implement some event tracking, I initially just littered Mixpanel Python functions across my backend codebase, only to have my Mixpanel project suddenly go missing after a few weeks. I didn’t really understand why, but I ended up settling on sending event data into my MongoDB database when they are triggered on my backend. What were once JIRA tickets I make to engineers for event tracking are now code that I implemented to capture these data. Event tracking never felt so real to me until now.
4. Client side event tracking — Unfortunately, I couldn’t figure out client side event tracking. It did help that I had both HTMX and JavaScript code on my frontend, and I was not sure how to send client side data for both types of interactions. This meant I couldn’t track some of the events that I wanted. This was another example of how my frontend engineering work, but more work would be needed if I need to meet the requirements of the data analyst me.

Dealing with NLB

When using the NLB API to get library information, I had to engage the NLB tech team about their APIs. Honestly, I expected better from them. Those interested can read my past posts here and here. It was quite a learning experience that was unfortunately painful at times. Eventually, I was glad that I managed to deploy my web app despite the odds.

My engineering side project also triggered another meeting with the NLB analytics team, which is a different team from the API team. This team had interviewed me previously about my scraping efforts to get book availability data from the NLB website, and this was before I knew the NLB API existed. This second meeting was more about my ideas on tech and data, and in relation to my experience using the NLB API.

Concluding thoughts

My engineering side project made me realise I was more passionate and competent in analytics than software engineering, which honestly isn’t a bad thing. For now, I most probably wouldn’t double down my efforts to become a full-stack engineer, as I think it would involve too much work and pull me too far away from my core analytics competencies.

That said, I did also gain some interesting perspectives on what a product team needs to do to build a workable tech solution, and it gives me a bit more confident to work with engineers, designers and product managers for my future analytics projects.

As for the web app, I do intend to make my app more user-friendly and visually pleasing once I have more time. In the meantime, Singapore residents who borrow physical library books, consider supporting a fellow Singaporean who hopes to make our society a better place with data and technology. This is its web link.

Fig 1 — My web app’s main page after login and saving some books

Thanks to everyone who has read this post. If you are interested in analytics side projects with a social science spin, follow me on Medium or Linkedin. Some topics I have explored include (1) Singapore housing prices and the updated Singapore million dollar public home analysis, (2) accessibility of Singapore hotels, (3) Taiwan housing prices, and (4) I even built a small web app for Singaporeans to track the library books they want to borrow. I also share less technical topics, like (5) how I learned to deal with uncertaintyand (6) how I end up being a freelance analytics consultant.

Lastly, I have a Substack (it is still alive) as well, where I share ideas on data concepts and strategies for targeted at busy business people.