Latest project: A Chrome extension that legally and ethically collects data from dynamic webpages

I was learning about natural language processing (NLP) through antivax posts from Facebook. I wanted to go deeper and analyze the comments, but faced a lot of challenges to get the comments. Automated bot like Selenium is out of question, as I don't want to risk my account or IP address getting banned.

I noticed that social media websites like YouTube and Facebook do not load all the data on its pages in one go. Instead, more data usually comes in as we scroll. I tried to right click the page to get its page source, but it turns out I only can see the initial codes and not the ones I received after scrolling.

There are specific platforms where you can pay for this kind of service. I saw someone selling their service, but the users have to pay seperately for each social media platform. Free tools are available, yet they tend to have traffic limits. For example, an hour it only offers 200,000 calls. So, if someone else decides to be selfish and use up the quota, then tough luck.

So yeah, I am building a Chrome extension. Since webscrapping is often illegal, I am building something that 'watches' the tab for new updates from the server, while the user browses the page. It works on YouTube and Facebook. In fact, to test the extension, I used it on a news website and manually browse the news all the way to January 2020, while it does the recording. Then, I took the data and combined with exchange rate and daily cases to build a dashboard, which I call Pandemic Monument.

I might make this an open source project. Not sure yet.

So far, I am the only user and I feel like doing more testing before releasing. But, feel free to let me know if you're interested.