Universal Video Scraper & AI Analysis Platform

Universal Video Scraper & AI Analysis Platform
https://github.com/sreejagatab/videoprocessing

Universal Video Scraper & AI Analysis Platform

A comprehensive, production-ready web application that automatically scrapes, downloads, processes, and performs AI-powered analysis on videos from any target URL.

🚀 Features

• Universal Video Support: Works with YouTube, Vimeo, Facebook, Twitter, TikTok, direct URLs, and 50+ platforms
• AI-Powered Analysis: Transcription, summarization, sentiment analysis, topic modeling, and more
• Real-time Processing: Live progress updates with WebSocket notifications
• Scalable Architecture: Handle viral content and traffic spikes
• Enterprise Ready: Production-grade security, monitoring, and reliability

🏗️ Architecture

Backend Stack

• FastAPI with SQLAlchemy and Alembic
• Celery + Redis for async video processing
• PostgreSQL for data storage
• MinIO/S3 for object storage
• FFmpeg for video processing
• yt-dlp for universal video downloading

AI/ML Stack

• OpenAI GPT for content analysis
• Whisper/AssemblyAI for transcription
• Transformers for NLP processing
• OpenCV for computer vision

Frontend Stack

• Next.js 14 with TypeScript
• Tailwind CSS for styling
• React Player for video playback
• SWR for data fetching
• Zustand for state management

🚀 Quick Start

Prerequisites

• Docker and Docker Compose
• Node.js 18+ (for frontend development)
• Python 3.11+ (for backend development)

Development Setup

1. Clone the repository

git clone <repository-url>
cd videoprocessing

2. Start the development environment

docker-compose up -d

3. Install dependencies

# Backend
cd backend
pip install -r requirements.txt

# Frontend
cd ../frontend
npm install

4. Run database migrations

cd backend
alembic upgrade head

5. Start the development servers

# Backend (in one terminal)
cd backend
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

# Frontend (in another terminal)
cd frontend
npm run dev

📁 Project Structure

videoprocessing/
├── backend/                 # FastAPI backend
│   ├── app/
│   │   ├── api/            # API routes
│   │   ├── core/           # Core configuration
│   │   ├── models/         # Database models
│   │   ├── services/       # Business logic
│   │   ├── tasks/          # Celery tasks
│   │   ���── utils/          # Utilities
│   ├── alembic/            # Database migrations
│   ├── tests/              # Backend tests
│   └── requirements.txt
├── frontend/               # Next.js frontend
│   ├── src/
│   │   ├── components/     # React components
│   │   ├── pages/          # Next.js pages
│   │   ├── hooks/          # Custom hooks
│   │   ├── utils/          # Utilities
│   │   └── types/          # TypeScript types
│   ├── public/             # Static assets
│   └── package.json
├── docker/                 # Docker configurations
├── k8s/                    # Kubernetes manifests
├── terraform/              # Infrastructure as Code
├── docs/                   # Documentation
└── docker-compose.yml      # Development environment

🔧 Configuration

Environment Variables

Create .env files in both backend/ and frontend/ directories:

Backend (.env)

DATABASE_URL=postgresql://user:password@localhost:5432/videoprocessing
REDIS_URL=redis://localhost:6379/0
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
OPENAI_API_KEY=your-openai-key
ASSEMBLYAI_API_KEY=your-assemblyai-key
SECRET_KEY=your-secret-key

Frontend (.env.local)

NEXT_PUBLIC_API_URL=http://localhost:8000
NEXT_PUBLIC_WS_URL=ws://localhost:8000

🧪 Testing

# Backend tests
cd backend
pytest

# Frontend tests
cd frontend
npm test

# E2E tests
npm run test:e2e

📊 Monitoring

• Health Check: GET /health
• Metrics: GET /metrics
• API Documentation: http://localhost:8000/docs

🚀 Deployment

Production Deployment

1. Build Docker images

docker build -t videoprocessing-backend ./backend
docker build -t videoprocessing-frontend ./frontend

2. Deploy with Kubernetes

kubectl apply -f k8s/

3. Or deploy with Docker Compose

docker-compose -f docker-compose.prod.yml up -d

📝 API Documentation

Interactive API documentation is available at:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

For support, please open an issue on GitHub or contact the development team.