Benwil VaultBenwil Vault
Start a project
← All workPRJ-004 · SaaS / Data Analysis Tool

Niche Analyzer

Client
Benwil Marketing (Internal Product)
Year
2026
Duration
2 months
Role
Founder
Overview

A full-stack data analysis platform designed to identify and evaluate profitable market niches using automated web crawling and interactive data visualization.

Niche Analyzer

Niche Analyzer is a powerful full-stack application designed for entrepreneurs, marketers, and data analysts to discover, evaluate, and categorize profitable market niches at scale. It seamlessly crawls lists of target websites, analyzes their content using AI, and streams actionable classification insights directly to an interactive dashboard in real-time.

Features

📊 Bulk Domain Processing

Upload lists of domains and let the system do the heavy lifting. Niche Analyzer efficiently processes bulk data to give you a comprehensive bird's-eye view of potential markets.

File Support

Easily upload .csv or .xlsx files containing raw URLs or domains.

Real-Time Streaming

Watch the analysis happen live. Results are streamed to the frontend row-by-row via Server-Sent Events (SSE), so you never have to wait for a massive batch to finish.

Queue Management

Pause, resume, and manage your analysis queue on the fly.

🧠 AI-Powered Classification (DeepSeek API)

Go beyond basic scraping. Niche Analyzer leverages advanced Large Language Models (LLMs) to intelligently categorize every website.

Site Type Detection

Automatically determines whether a website is an Ecommerce, Service, Blogger, News, or Generic site using deep content analysis combined with hybrid rule-based heuristics such as cart and checkout flow detection.

Niche Extraction

Identifies both the Primary Niche (Niche 1) and Secondary Niche (Niche 2) for every website, complete with AI confidence scores.

Content Generation

Automatically detects whether a website contains a home blog and generates three actionable blog post ideas tailored specifically to that website's niche.

Language Detection

Automatically identifies non-English websites, allowing them to be skipped to reduce processing time and save AI API usage.

🕸️ Intelligent Web Crawling

A highly configurable crawling engine built to efficiently handle both traditional and modern websites.

Dynamic Rendering

Optional Playwright integration enables JavaScript execution, allowing the crawler to fully analyze Single Page Applications (SPAs) and other dynamic websites.

Budget Controls

Configure crawl limits including:

  • Maximum Pages per Website

  • Maximum Crawl Time

  • Per-Page Timeout

  • Crawl Concurrency

Proxy Support

Built-in HTTP(S) proxy support helps distribute requests and reduce blocking during large-scale crawling.

Polite Scraping

Supports configurable crawling behavior, including:

  • Respecting robots.txt

  • Configurable cooldown periods between websites

  • Resource-friendly crawling

Installation (Local Development)

Step 1 — Clone the Repository

Clone or download this repository to your local machine.

Step 2 — Navigate to the Backend

cd backend

Step 3 — Install Dependencies

npm install

Step 4 — Configure Environment Variables

Copy the environment template.

cp .env.example .env

Add your DeepSeek API key to the .env file.

Step 5 — Install Playwright (Optional)

If you plan to crawl JavaScript-rendered websites, install the Playwright browsers.

npm run playwright:install

Step 6 — Start the Backend Server

npm start

The backend server runs on:

http://localhost:4000

Step 7 — Start the Frontend

Open another terminal.

Navigate to the frontend directory, install dependencies, and start the Vite development server.

npm install
npm run dev

How to Use

Step 1

Open the Niche Analyzer dashboard in your browser.

Step 2

Click Upload and select a CSV or Excel file containing the URLs you want to analyze.

Step 3

Configure your crawl settings, including:

  • Concurrency

  • Maximum Pages

  • Playwright

  • Timeout

  • Proxy

  • Crawl Budget

Step 4

Select the rows you wish to process and click Start Analysis.

Step 5

Watch the dashboard populate in real time as the crawler extracts website data and the DeepSeek AI classifies each website.

Step 6

Export the enriched dataset once processing is complete for marketing, SEO, lead generation, or research purposes.

Architecture

backend/src/server.js

The core Express.js server responsible for:

  • File uploads using Multer

  • API route definitions

  • Server-Sent Events (SSE)

  • Live frontend communication

backend/src/worker/

Responsible for:

  • Concurrent job processing

  • Queue management using p-queue

  • Crawl orchestration

  • AI classification workflow

backend/src/crawler/

The website extraction engine built with:

  • Axios

  • Cheerio

  • Playwright

Responsible for:

  • Static HTML extraction

  • JavaScript rendering

  • Website content collection

frontend/

A React single-page application built using:

  • React 18

  • Vite

  • Tailwind CSS

  • Framer Motion

  • TanStack Table

Provides an interactive dashboard for monitoring live analysis results and managing processing jobs.

Technologies Used

Backend

  • Node.js

  • Express.js

  • Axios

  • Cheerio

  • Playwright

  • p-queue

Frontend

  • React 18

  • Vite

  • Tailwind CSS

  • TanStack Table

  • Framer Motion

AI & Integrations

  • DeepSeek API

Data Handling

  • Multer

  • xlsx

  • csv-parse

Results