Lorem ipsum dolor sit amet consectetur adipisicing elit. Architecto, aliquam pariatur. Illum officiis nam dolore eaque nostrum tenetur necessitatibus, nobis exercitationem rem laborum odio molestiae doloribus perferendis, voluptas quia officia porro veniam aperiam recusandae, minima commodi?
Web crawling at scale is essential for aggregating data, monitoring websites, and enabling downstream applications like machine learning models and analytics. This project involved developing a scalable web crawling and re-crawling pipeline using Scrapy, storing the data in MongoDB, and deploying a machine learning model accessible through a Flask API. The entire pipeline was Dockerized and deployed on AWS EC2 for robust, high-availability operations.
