Ecommerce price crawler

Project Title

Ecommerce Price Crawler

Project Description

Abrar Balti required the development of a backend system for a web scraping project. The frontend was already completed, and the main task involved creating the backend using Django, with a focus on web scraping and microservice architecture.

Key Responsibilities

Backend Development:
- Developed the backend for a web scraping project using Django.
- Implemented microservice architecture to ensure scalability and maintainability.
- Developed APIs for communication between different microservices.
Web Scraping:
- Used Python libraries such as Requests and Beautiful Soup to scrape data from various e-commerce websites.
- Stored the scraped data in a MongoDB database.
- Ensured that the scraping process is efficient and handles errors gracefully.
Microservice Implementation:
- Created independent microservices for different operations such as fetching data, searching products, etc.
- Ensured that each microservice is independently deployable and communicates with others via APIs.
Database Integration:
- Stored the scraped data in a MongoDB database.
- Ensured that the database schema is well-designed and supports the required queries efficiently.
Client Communication:
- Maintained clear and frequent communication with the client.
- Provided regular updates on the progress of the project.
- Ensured that the client’s requirements are met and that any changes are communicated and agreed upon.

Technologies Used

Language: Python
Framework: Django Microservice Architecture
Frontend: HTML, CSS & Javascript
Web Scrapping: Beautiful Soup
Database: MongoDB

Challenges Faced:

Budget Constraints: The client had a limited budget, which required efficient time and resource management.
Microservice Architecture: Implementing microservices for the first time required additional research and learning.
Tight Deadline: The project had to be completed before the client’s presentation date, requiring effective time management and prioritization of tasks.

Results Achieved

Successfully developed a backend system using Python and Django.
Implemented web scraping functionality for multiple e-commerce websites.
Created a microservice architecture that ensured scalability and maintainability.
Integrated the system with MongoDB for efficient data storage and retrieval.
Delivered the project within the given budget and timeframe, meeting all client requirements.