Data sheet on the growth of paid stripe users on 10,000 websites in August-September 2024

In August, I noticed a gap in the market for comprehensive data on paid user growth across various software products. Realizing the potential value of such information, I embarked on a challenging two-month journey to compile and analyze this data.

My first obstacle was data collection. I developed a custom web scraping tool using Python and Selenium to gather information from thousands of websites. This process was complex due to varying website structures and anti-scraping measures. I had to implement IP rotation and user agent switching to avoid being blocked.

Next, I faced the challenge of identifying Stripe users among these websites. I created a fingerprinting algorithm that could detect Stripe integration patterns in website source code. This required extensive testing and refinement to achieve high accuracy.

Data cleaning and normalization were the next hurdles. I wrote a series of Python scripts to standardize the collected data, remove duplicates, and fill in missing information where possible. This step was crucial for ensuring data consistency and reliability.

To track growth over time, I set up a database using PostgreSQL and implemented a daily update routine. This allowed me to capture changes in paid user numbers and calculate month-over-month growth rates.

One of the most significant challenges was estimating paid user numbers for websites that didn't publicly display this information. I developed a machine learning model trained on known data points to make educated guesses based on various website metrics and characteristics.

After overcoming these technical challenges, I finally compiled a comprehensive dataset covering approximately 10,000 companies. The data includes:

1. Website name

2. Website URL

3. Estimated paid user count

4. Month-over-month growth rate

5. Industry category

This dataset now serves as a valuable resource for entrepreneurs, product managers, and analysts looking to understand market trends and identify successful business models.

I'm continually working on improving the accuracy and breadth of this data. I welcome feedback from users to help refine the collection and analysis processes further.