How to develop a side project

This post is the first in a series of articles where I explain how I had an idea for a side project, I thought: it can’t be that hard to build. And almost a year later, I have an open source Github repo, my app is deployed and works automatically without me doing anything.

In this series I will tell you how I did it.

This first post is a general post explaining in broad terms how to develop an idea into a side project, the challenges, how to fix them, and personal tips.

0. The idea

You can start a side project about anything, there’s even webpages with side project ideas that you can borrow. These can be good to practice but if you’re thinking of starting a project to work on for at least some months, then I recommend that you choose a topic that interests you.

Times goes faster when we do things that we enjoy

For the sake of brainstorming, what would be a cool website, app, tool or analysis that you would personally be interested in? It doesn’t matter if you have no idea how to do it, just think of: what would you like to see in the world that doesn’t exist right now?

Some example ideas:

• An app where you can write the workouts that you do, and every day it reminds you to do them. It can keep streaks, increase difficulty as you go along, allow for free days, etc. I’m currently working on something like this at the moment 😄
• How much do political parties engage on social media, and the comparison between different parties from the same country. Maybe that is related to election results? Just today I saw a journalist post this data on Twitter regarding German elections.
• A website showing the 7-day-incidence throughout months at a specific location. Someone I know did a website for Berlin
• How spreaded are COVID conspiracy theories on Twitter? Someone I know did a conspiratorial tweet detection app

There can be countless examples, for these two specific people that I know, they had no idea how to build their apps when they started. They had the motivation to spend time on it, they worked hard to get the data, and worked even harder on the development.

A project to learn and build.

A months-long project should be a mixture of learning and developing. Writing code for something you already know how to do is monotonous and you will get bored. If you really know the technology, then you will finish fast and you can move to other things. But in this series, I’m referring to a project where you want to learn and build.

Think big. If you’re motivated to build it, you will manage to learn it.

Of course, if the technical specifications are very far away from what you can do, you will have to spend more time learning and the project will take more time.

For example: if you want to analyze tweets and you only know basic Python, you will have to learn how to interact with the Twitter API, text processing, and maybe some NLP.

Make a list of the knowledge that you need and you don’t know for the project. I usually follow this criterion:

I accept an idea for a project if it has 2 new things that I will have to learn.

One new thing is not motivation enough, and more than two is too much for me. If I had to build an authentication service with cybersecurity on Ruby, it would be too challenging for me because I know none of those things and I would get stuck too often, which would lead to frustration and losing motivation. This has happened to me several times.

To conclude this part, pick an idea that interests you, that you don’t know how to do but you feel like you can find out.

How I found my idea.

In my case, I was at a rock climbing gym I like to go to. It gets very busy on weekends and in the evenings during the week, so the company put a kind of traffic light on the website where it says how full it is. You can check the website before you go and if it is too full, you might want to go some other time.

This day, I had checked the traffic light before going, it said it was mostly empty. But when I arrived, it was super full! I got annoyed and I thought the current number was not useful. If it says 80% full, how do I know if it was 90% full 10 minutes ago and now people are leaving, or the contrary, it was 50% before and now so many people are coming? In the first scenario I might go to the gym and in the second I wouldn’t.

If only they saved past data and show a kind of graph. I thought, it would be 3 easy steps: get the data from the website every 20 minutes or so. Save it somewhere. Have a website taking the saved data and showing a graph. I remember thinking:

It can’t be that difficult.

This is similar to what Marshall Eriksen from How I met your mother says: “I can walk that far”. As you can imagine, the distance is much bigger than he expected and he indeed cannot walk that far. 😄

When I sat down at my computer and started coding the part to get the data from the website, I realized it was not so easy. Now after a year I can say it is easy for me, because I have already done it and I know how to do it. If I had to do something similar to this project again, it would take me so much less time. But this is thanks to everything I learned while I struggled. At the time it was really difficult.

1. Learning

Photo by Green Chameleon on Unsplash

The part of getting the data from the website was called “scraping”, a word I knew but I had never done it. After some research, I found that there are several libraries in Python for scraping, Basically you make a request to the website, it returns the HTML as a long string and you have to find your information there, usually with Regex.

Just doing this part requires knowledge in Python, a bit of front-end and Regex. Thankfully I was already familiar with these concepts so once I knew how the library worked, finding the information was straighforward.

Throughout the project however there were many concepts I wasn’t familiar with and these took me a long time to learn. I spent countless hours reading tutorials and watching videos. This is frustrating because you just want to work on your project, not watch that video or read that article. But the knowledge you acquire in this step will empower you to develop this particular project, and other similar projects in the future.

If you get stuck in your learning, you might have to search in the second page of Google, or ask a question to the community on Stack Overflow or Reddit.

It might happen that you can’t advance in your learning because when you read about technology X, it is based on technology Y which you also don’t know. Then you go and read about Y, and you find out it is based on technology Z, and so on. This can be very frustrating but it’s a good kind of frustrating because you are learning. Once you find the root, the basic concept or technology that you understand, you can go back the chain until you finally understand technology X.

For instance, this happened to me with AWS Lambda. This is a service from the AWS cloud provider, which means that the code runs in the cloud and not in your computer. I used this to scrape the websites every 20 minutes. You write the scraping code (the requests and the regex), the code to save the information, and then you need to add a specific format and specific commands so you can deploy your code to AWS Lambda.

I already had the scraping code and the way to save the information, but I had no idea how to connect it with AWS Lambda. I spent lots of hours reading, first you need an AWS account, if you want your code to run every 20 mins you need to write it in a specific way for AWS, then you can deploy your code manually but that’s not ideal so I looked for an automated approach, that made it even more difficult. I found a framework to deploy automatically and it took me a long time to learn how everything worked together.

2. Debugging

Photo by Nubelson Fernandes on Unsplash

At some point, you will find the way to implement what you want. You will copy the code from a website, paste it in your code editor and run it. Most probably, it won’t work. And most probably, you won’t know why. Debugging something you don’t know is very difficult.

First of all, read the error carefully. This is good software development practice, many times the error message itself tells you how to fix it. It will tell you in what line of the code it failed, so you can narrow down. The error type also helps a lot. Is it a programming language error? For instance bad formatting, undefined variables, wrong variable types, a library that you forgot to install. These errors happen all the time so it’s important to understand them so you can fix them fast.

If the error refers to the new code you just pasted, first focus on understanding what your code does. What you just pasted, what does each part do? Why? How do the different parts link together? The variables for the first part match the variables for the second part? Spend some time researching what the code does. Maybe you initialize something, but first you need to set up an account. Or you’re trying to use a library but you need to initialize something else first.

It might be the case that while you research this, you come across the problem you have and you are able to solve it. Or you read about some other information that helps you with the problem.

If not, copy and paste the error in Google. This will probably show you Stack Overflow or Github pages with the same, or a similar problem. Something I personally do wrong is that I don’t even read the problem, I just directly look for the solution and quickly read it and copypaste it into my code. When a code snippet works, then Icheck where I got it from and I try to understand it.

Ideally, you should read the text in all the pages. Someone posted a question in Stackoverflow or Github with the same problem as you or a smilar one, so read about their scenario. Then read the answers, what the person explains, the code, and any other answers. This will allow you to learn how to fix your problem and additionally, how to fix similar problems.

2.1. Asking in the forums (Stack Overflow, Github)

If you can’t find your problem online or you did but none of the solutions work, consider writing in the forums yourself. Ask a question, or answer to a solution saying that it didn’t work, and the error you got. People in the forums are usually helpful and happy to help.

Personally, I like to exhaust all other options before asking in the forums. I just can’t believe that I’m the first person in the world to have this issue so I try to find anyone else who already posted about this. However, sometimes there is just no way, so I decide to ask in Stack Overflow. But then, something interesting happens.

I try to write my question as detailed as possible, explaining my problem, what I tried, what didn’t work, all the ways I tried to make it work and the error it showed when it failed.

I try to think of what people will ask me in the responses. What if the solution is really simple? While I’m writing my question, ideas come to mind: what if I try this? I will try and if it doesn’t work, I will include it in my question.

It has already happened to me 4 or 5 times that I have almost finished writing my Stack Overflow question, I’m ready to press Publish and I think: what if I try this last thing? And then it works. This moment is magical.

However if you do end up writing a Stack Overflow question, make it as thorough as possible, write all the things you’ve tried, write the question in a clear way that explains your problem clearly, and don’t forget to add appropriate tags. This way, people who are looking to answer questions can find your question easier and you will reach more people.

If all fails, reach out to people you know in real life. Maybe they don’t know about your particular problem but they can advise you things to try that you didn’t even think of.

3. Version control

At this point, hopefully you have your code working. It took a long time, you had to learn about a new technology, found the code and made it work.

It’s very very frustrating when after all this work, you continue developing your project and suddenly your code doesn’t work anymore, and again, you don’t know how to fix it. That’s why version control is crucial.

With version control, you can save your versions. Whenever you add a new piece of code and it works, you can commit it. Then you have a version of your code that works. When you add something else, the version control tool shows you the changes you’ve done since the last version and you can see if you changed something you shouldn’t have. Github and Gitlab are two websites you can use to manage your code.

4. Learning, developing, debugging

Throughout your project you will go around in the circle of learning and debugging, until you develop a project with code that you understand. Over time, the learning time will decrease, the developing time will increase and you won’t have to debug as much.

5. Showcasing your project

When I was developing my project, I made an effort to write about it on my Twitter account. The struggles I was having, the things I was learning. It’s very inspiring to learn about a project when it’s finished but I believe it also makes it look easy when most probably it hasn’t. That’s why I tried to show the work in progress as well.

When finished, I wrote about it on Twitter as well and I signed up to talk about my project in a programming meetup. This made more people know about my project, and actually I received some very interesting advice about how to improve my code! Additionally, some people volunteered to participate in my project as well!

Conclusion

In this post I explained how to develop an idea into a proper side project. It starts with an ambitious idea, lots of learning, even more debugging and slowly with time and patience, building a side project you can show in your social networks and even talk about it on a meetup.

I hope you liked this post, and stay tuned for the next one! Follow me on Twitter for more stories :)