CargoVision AI

• About CargoVision AI
  • The Problem
  • The CargoVision AI Solution
  • Operational Scope
• Screenshots and Demo
  • Application Dashboard
  • Inspection Workspace and Results
  • Operational Telemetry and Analytics
  • Wagon Detection and Stencil Localization
  • Structural Defect Detection
  • Deep Image Restoration
  • Evaluation Video Footage
• Key Features
• Computer-Vision and OCR Pipeline
  • Verified Checkpoint Inventory
  • Cascaded Processing Stages
• System Architecture
  • Pipeline Flow Diagram
  • System Component Workflow
  • Inference Processing Workflow
• Technology Stack
• Prerequisites and System Requirements
  • Software Prerequisites
  • Hardware Sizing Guidance
• Installation and Local Setup
  • Windows Setup (PowerShell)
  • Linux and macOS Setup (Bash)
• Running CargoVision AI
  • 1. Start Backend API
  • 2. Start Frontend UI
  • 3. Access Local Endpoints
• Configuration and Environment Variables
• API Reference
• Database, History and Reports
  • SQLite Relational Schema
  • Compliance PDF Dossier Generation
• Testing and Verification
• Performance and Empirical Benchmarks
  • Accuracy Metrics
  • Hardware Footprint (Measured on CPU)
• Privacy, Security and Deployment
• Limitations and Roadmap
• Contributing
• License and Attribution

Railway freight yards are critical logistics arteries. Millions of metric tons of raw materials, containers, and bulk commodities move daily across vast rail networks. However, rolling stock inspection has historically relied on physical visual examination by yard inspectors walking trackside with handheld clipboards:

• Human Inspection Bottlenecks: Manual examination of 50+ wagon rakes takes hours and exposes field personnel to hazardous trackside conditions.
• Environmental Impediments: Dust, low-light tunnel transits, twilight operations, and motion blur from moving consists severely degrade stencil legibility.
• Transcription Errors: Missing or misread wagon markings cause logistical routing anomalies, wagon misplacement, and delayed consist dispatch.

CargoVision AI is a specialized computer-vision inspection engine designed to automate trackside rolling stock monitoring. The system ingests high-resolution video streams from trackside cameras, executes cascaded deep learning inference, extracts and validates 11-digit rolling stock identification stencils, flags structural anomalies (such as broken suspension springs or axle cover defects), and automatically compiles executive PDF compliance reports.

The following visual showcase highlights actual application interfaces, model outputs, and computer-vision components present directly in the repository:

Figure 1: Main CargoVision AI web dashboard featuring trackside engine telemetry, interactive console monitoring, and system navigation in the dark-charcoal and lime-accented interface.

Figure 2: Freight Inspection Console workspace showing completed video processing (Run #1), multi-stage pipeline status tracking, telemetry summary (total wagons, inference speed, lux), and audit report generation trigger.

Figure 3: Operational telemetry panel displaying aggregate system volume metrics, real-time multi-model FPS, Indian Railways checksum pass fidelity, and a multi-run wagon volume vs. defect distribution chart.

Wagon Bounding Box Detection	Number Stencil Localization & OCR
Figure 4A: YOLOv8 detector localizing individual freight wagons and tracking consistent inventory IDs across camera frames.	Figure 4B: Isolated number stencil crop feeding EasyOCR character recognition and Modulo-10 checksum parsing.

Suspension Spring Damage Classification	Axle Cover Fault & Missing Bolt Detection
Figure 5A: Defect classifier identifying intact springs ("Good") versus displaced/fractured suspension coils ("Bad").	Figure 5B: Structural inspection identifying intact axle covers versus missing or sheared retaining bolts.

Figure 6: Side-by-side comparison of degraded low-light wagon footage before (left) and after (right) adaptive Zero-DCE curve estimation and NAFNet edge enhancement.

The repository includes real sample railway video clips in Assests/ for reproducible local evaluation:

• Automated Rolling Stock Detection & Counting: Custom-trained YOLOv8 models localize wagons as they enter the frame, assigning unique sequence IDs to prevent duplicate counting.
• 11-Digit Number Stencil Localization & OCR: Dedicated YOLOv8 sub-detector pinpoints the stencil badge; EasyOCR extracts alphanumeric strings; an Indian Railways parser validates the standard Modulo-10 checksum: $$\text{Checksum} = \text{Mod}_{10}\left(\sum w_i \times d_i\right)$$ Decodes wagon type (Digits 1–2), owning railway zone (Digits 3–4), manufacturing year (Digits 5–6), and serial number (Digits 7–10).
• Adaptive Low-Light & Deblur Restoration: Zero-DCE dynamic curve estimation restores underexposed frames ($< 60$ mean brightness), while NAFNet deblurring sharpens motion-blurred number crops when Laplacian variance falls below threshold ($< 100$).
• Structural Defect & Anomaly Telemetry: YOLOv8 damage detector flags unlatched hopper/box doors, fractured suspension springs, and axle cover defects, linking anomalies directly to specific wagon IDs.
• Searchable SQLite Audit Ledger: Every inspection run logs immutable telemetry, wagon crop paths, blur scores, and defect counts into an embedded relational database.
• Dynamic PDF Dossier Generation: One-click generation of multi-page PDF compliance reports with embedded image crops and verification badges using FPDF2.

All 8 active model weights exist directly in the repository on branch cargovision-main (total size: 147.78 MB; no file exceeds 65.5 MB):

1. Ingestion & Conditioning: Raw video frames are analyzed for luminance. If average brightness is $< 60$, Zero-DCE applies 8 progressive iterations of pixel-wise quadratic curve enhancement.
2. Wagon Localization & Inventory Tracking: The primary YOLOv8 model localizes wagons and tracks bounding boxes across frames using spatial IoU association.
3. Number Stencil Crop: A specialized YOLOv8 head detects the stencil coordinates on the wagon body and crops the number region.
4. Motion Deblurring: If the crop's Laplacian blur variance is $< 100$, NAFNet restores sharp edges prior to text recognition.
5. OCR & Algorithmic Validation: EasyOCR extracts characters; the Indian Railways parser checks format validity and computes the Modulo-10 checksum digit.
6. Defect Telemetry & Report Compilation: Detected anomalies (open doors, spring faults) are linked to wagon IDs, saved to SQLite, and prepared for PDF export.

Figure 7: Detailed deep learning architecture showing the dual-path illumination routing (Zero-DCE / NAFNet), YOLOv8 detection backbone, image recognition head, and ConvLSTM2D damage classifier. Technical documentation is also available in Architecture pdf.pdf.

flowchart TD
    subgraph Client["Frontend Client (React 19 + TypeScript + Vite)"]
        UI["Modern Nuxt UI Dashboard"]
        UP["Upload & Video Feed Engine"]
        HIST["History, Audit & Search View"]
        PDF_DL["PDF Dossier Downloader"]
    end

    subgraph Server["Backend Server (FastAPI + Python 3.11)"]
        ENTRY["run.py / Uvicorn Server (Port 8000)"]
        SEC["Upload Validator & Concurrency Ceiling"]
        QUEUE["FastAPI Background Queue"]
        ROUTER["REST Endpoints & MJPEG Streamer"]
    end

    subgraph Vision["Cascaded Vision Pipeline"]
        PRE["Preprocessing: CLAHE & Zero-DCE"]
        W_DET["YOLOv8 Wagon Detector"]
        D_DET["YOLOv8 Damage & Door Detector"]
        N_DET["YOLOv8 Number Localizer"]
        DEBLUR["NAFNet Deep Deblurrer"]
        OCR["EasyOCR 11-Digit IR Parser"]
    end

    subgraph Persistence["Storage & Artifacts"]
        DB[("SQLite Database (inspections.db)")]
        CROPS["Wagon BBox Crop Cache"]
        REPORTS["Generated FPDF2 Inspection Reports"]
    end

    UI -->|HTTP / REST| ROUTER
    UP -->|Multipart Video Upload| SEC
    SEC --> ENTRY
    ENTRY --> ROUTER
    ROUTER --> QUEUE
    QUEUE --> Vision
    Vision --> Persistence
    Persistence --> ROUTER
    ROUTER --> HIST
    ROUTER --> PDF_DL

flowchart TD
    A["Raw Video Ingestion (.mp4 / .avi / .mov)"] --> B{"Average Frame Brightness < 60?"}
    B -- Yes --> C["Zero-DCE Curve Enhancement / SCI"]
    B -- No --> D["Standard Preprocessing (CLAHE)"]
    C --> E["YOLOv8 Wagon Detection (merged_model_v6)"]
    D --> E
    E --> F["Extract Bounding Box & Track Wagon ID"]
    F --> G["YOLOv8 Damage Detector (damage_detector_v1)"]
    G --> H{"Defect Flagged? (Door / Spring / Axle)"}
    H -- Yes --> I["Log Defect Class & Confidence"]
    H -- No --> J["Mark Wagon Structurally Intact"]
    F --> K["YOLOv8 Number Localizer (number_detector_v1)"]
    K --> L{"Laplacian Variance < 100?"}
    L -- Yes (Motion Blur) --> M["NAFNet GoPro Width-32 Restoration"]
    L -- No (Sharp) --> N["Direct Crop Extraction"]
    M --> O["EasyOCR Text Recognition (Alphanumeric Whitelist)"]
    N --> O
    O --> P["Indian Railways 11-Digit Checksum Validator"]
    P --> Q{"Modulo-10 Checksum Valid?"}
    Q -- Valid --> R["Store Verified 11-Digit Rake Identification"]
    Q -- Invalid --> S["Store Raw OCR String with Validation Warning"]
    I --> T["Write Inspection Record to SQLite"]
    J --> T
    R --> T
    S --> T
    T --> U["Compile Executive PDF Compliance Dossier"]

• Operating System: Windows 10/11 64-bit (PowerShell 5.1 or 7+), Linux (Ubuntu 22.04+ LTS), or macOS 13+.
• Python: Python 3.11.x (Verified on Python 3.11.9).
• Node.js: Node.js v18.0.0 or higher (Verified on Node.js v24.14.0, npm 11.20.0).
• Git: Git 2.40+ (Verified on Git 2.45.2).

• CPU: Quad-core Intel/AMD processor minimum (8+ cores recommended for faster multi-threaded video decoding).
• System RAM:
  • Minimum: 8 GB RAM.
  • Measured Idle RAM (Models Loaded): 579.80 MB.
  • Measured Peak RAM (1080p Video Inference): 744.13 MB.
  • Recommended: 16 GB RAM for comfortable multitasking.
• GPU (Optional): NVIDIA GPU with CUDA 11.8 / 12.1 support. Speeds up inference to 25–40 FPS. CPU execution is fully supported (averaging ~5.09 FPS on an 8-thread CPU).
• Disk Space: ~5 GB free space for repository code, model weights, virtual environment, and dependencies.

Open PowerShell and execute the following commands:

# 1. Clone the repository on branch cargovision-main
git clone -b cargovision-main https://github.com/ayushcmd/CargoVision-AI.git
cd CargoVision-AI

# 2. Create and activate a Python virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1

# 3. Upgrade pip and install backend dependencies
python -m pip install --upgrade pip
pip install -r deployment/requirements.txt

# 4. Install frontend dependencies
cd frontend
npm install
cd ..

# 5. Copy environment templates
Copy-Item .env.example .env
Copy-Item frontend\.env.example frontend\.env

(If PowerShell blocks script activation, run: Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass)

# 1. Clone the repository on branch cargovision-main
git clone -b cargovision-main https://github.com/ayushcmd/CargoVision-AI.git
cd CargoVision-AI

# 2. Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# 3. Install backend dependencies
pip install --upgrade pip
pip install -r deployment/requirements.txt

# 4. Install frontend dependencies
cd frontend
npm install
cd ..

# 5. Copy environment templates
cp .env.example .env
cp frontend/.env.example frontend/.env

To run CargoVision AI locally, open two terminal windows:

From the project root directory:

# Activate Python virtual environment
.\.venv\Scripts\Activate.ps1

# Start the FastAPI backend via run.py entrypoint
python run.py

The backend initializes the SQLite database, preloads models, and binds to http://127.0.0.1:8000.

From the project root directory:

cd frontend
npm run dev

The Vite dev server compiles the React 19 SPA and serves it at http://127.0.0.1:5173.

Once both services are running, open your browser:

All configuration parameters use sensible defaults for local development:

CargoVision AI provides a fully documented RESTful API. Below are the verified primary endpoints:

• GET /health — Health check and telemetry probe. Returns service version, active jobs, and database status.
• GET /docs — Interactive OpenAPI Swagger UI.
• GET /openapi.json — Raw OpenAPI schema.

• POST /upload
  • Purpose: Upload a video for cascaded computer vision processing.
  • Content-Type: multipart/form-data (Field: file).
  • Accepted Formats: .mp4, .avi, .mov, .mkv, .webm (Max: 35 MB).
  • Response (200 OK): {"message": "Upload successful...", "inspection_id": 1, "video_name": "..."}.
  • Error (429 Too Many Requests): Returned if another video is currently actively processing.
• GET /inspections/{inspection_id}/status
  • Purpose: Poll inference progress and state (PROCESSING, COMPLETED, FAILED).

• GET /history — Retrieve all historical inspection runs stored in SQLite.
• GET /history/{inspection_id} — Fetch detailed detection records, wagon crops, and defect breakdowns for a specific run.
• GET /history/{inspection_id}/report — Download dynamically compiled FPDF2 PDF compliance dossier (application/pdf).
• GET /stats — Aggregate operational metrics (total wagons inspected, defect counts, processing speeds).
• POST /api/search — Search inspections by wagon number, date range, or anomaly class.

• GET /video_feed/{stream_id} — Live multipart MJPEG video stream with bounding-box canvas overlay.
• POST /live/start & POST /live/stop — Controls trackside camera RTSP capture loops.

CargoVision AI manages its relational schema via full model/src/core/database.py upon startup:

┌─────────────────────────────────┐
│          inspections            │
├─────────────────────────────────┤
│ id (PK, AutoIncrement)          │
│ video_name (TEXT)               │
│ timestamp (TEXT)                │
│ total_wagons (INTEGER)          │
│ enhanced_video_path (TEXT)      │
│ status (TEXT)                   │
│ fps (REAL)                      │
│ resolution (TEXT)               │
│ avg_brightness (REAL)           │
│ blur_stats (TEXT, JSON)         │
└────────────────┬────────────────┘
                 │ 1:N
                 ▼
┌─────────────────────────────────┐       ┌─────────────────────────────────┐
│        detection_results        │       │          system_audit           │
├─────────────────────────────────┤       ├─────────────────────────────────┤
│ id (PK, AutoIncrement)          │       │ id (PK, AutoIncrement)          │
│ inspection_id (FK)              │       │ event_type (TEXT)               │
│ wagon_number (TEXT)             │       │ details (TEXT)                  │
│ confidence (REAL)               │       │ timestamp (TEXT)                │
│ is_damaged (BOOLEAN)            │       └─────────────────────────────────┘
│ defect_type (TEXT)              │
│ image_path (TEXT)               │
│ timestamp (TEXT)                │
└─────────────────────────────────┘

When an inspection completes, full model/src/core/report_generator.py uses FPDF2 to generate a standard compliance dossier:

• Header block with inspection ID, timestamp, and video filename.
• High-level telemetry summary (total wagons, defect ratio, average brightness).
• Tabular breakdown of each wagon: Wagon Index, Extracted 11-Digit Number, Modulo-10 Checksum Validity Badge, Damage Classification.
• Embedded visual crop evidence for every wagon and anomaly flagged.

CargoVision AI includes automated verification suites:

Tests health endpoints, OpenAPI routes, invalid file extension rejection (.exe), and upload size limits:

python deployment/test_security.py

Result: 5/5 PASSED.

Verifies API availability, database integrity, real video upload, and concurrent request protection:

python deployment/smoke_test.py

Result: PASSED.

Validates TypeScript typings, component routing, and Vite asset bundling:

cd frontend
npm run build

Result: Clean compilation (dist/ index.html, CSS, and JS generated).

Based on reproducible benchmarks run on local hardware (Intel CPU, 8 threads, 16 GB RAM) documented in evaluation/reports/accuracy_benchmark.md and deployment/measured_resources.json:

• Wagon Detection mAP@0.50: 0.942 (Baseline: 0.924, +1.9% improvement).
• Number Localization mAP@0.50: 0.897 (Baseline: 0.865, +3.7% improvement).
• 11-Digit OCR Recognition Accuracy: 82.1% correct parses across benchmark rail sequences.
• Structural Defect Recall: 91.3% across open doors, deformed side panels, and spring faults.

• Base Python Process: 34.4 MB RAM
• Import Footprint (Torch + FastAPI + OpenCV): 315.6 MB RAM
• Idle Memory (All Models Pre-Loaded): 579.80 MB RAM
• Peak Memory During 1080p Video Inference: 744.13 MB RAM
• Inference Speed: ~5.09 FPS on CPU (~25–40 FPS on CUDA GPU).

• 100% On-Premise Data Sovereignty: Video files, extracted crops, and audit records remain entirely on your local machine. No telemetry or imagery is transmitted to third parties.
• Upload Validation & Hardened Extensions: The /upload endpoint strictly validates file extensions against .mp4, .avi, .mov, .mkv, .webm and enforces a 35 MB payload limit.
• Concurrency Serialization: Video processing is CPU-intensive. A single-job lock (MAX_CONCURRENT_JOBS=1) serializes requests, returning HTTP 429 Too Many Requests to prevent system thrashing or out-of-memory crashes.
• Zero Hardcoded Secrets: Training scripts have been sanitized to use os.environ.get("ROBOFLOW_API_KEY", ""). No passwords, tokens, or credentials are baked into the Git history or frontend bundles.

For users seeking to share a public demo of their local CargoVision AI engine without paying for cloud servers:

1. Frontend: Deployed to Cloudflare Pages for global CDN availability at $0/month.
2. Backend: Running locally on your computer, exposed securely via a free Cloudflare Tunnel (cloudflared):

   .\cloudflared.exe tunnel --url http://127.0.0.1:8000

3. Set VITE_API_BASE_URL in Cloudflare Pages to your public tunnel URL.

• Cost: Strictly $0.00 / month (₹0.00 INR).
• Credit Card Required: None.

(Detailed runbooks and hosting provider feasibility studies are documented in deployment/FREE_HOSTING_COMPARISON.md and deployment/DEPLOYMENT_RUNBOOK.md).

• Inference Throughput: CPU inference averages ~5 FPS, requiring hardware GPU acceleration (CUDA) for real-time 30+ FPS processing on high-speed rail lines.
• Extreme Dirt / Obscured Stencils: Severely weathered or paint-scratched stencils may fail Modulo-10 checksum validation, logging raw text with warning flags.
• Trackside Lighting: Extreme direct headlight glare can temporarily oversaturate camera dynamic range before CLAHE normalization.

• Core Cascaded Pipeline: YOLOv8 wagon detection, number localization, and damage detection.
• Adaptive Image Restoration: Integrated Zero-DCE low-light enhancement and NAFNet deblurring.
• Indian Railways 11-Digit Parser: Modulo-10 checksum algorithm verification.
• Premium Nuxt UI Frontend: Responsive React 19 + Vite dashboard with live metrics and stream viewer.
• Automated PDF Dossier Generator: Dynamic multi-page PDF inspection reports with embedded crops.
• ONNX Runtime Export: Quantize PyTorch models to INT8 ONNX for 2x faster CPU inference.
• Multi-Camera Parallel Feeds: Simultaneous dual-track side-view camera ingestion.
• Hardware Acceleration: Native TensorRT execution profiles for NVIDIA Jetson trackside edge devices.

Contributions, feature proposals, and bug reports are welcome!

1. Fork the repository: https://github.com/ayushcmd/CargoVision-AI
2. Create a feature branch: git checkout -b feature/improved-ocr-parser
3. Commit your changes: git commit -m "feat(ocr): improve character segmentation under glare"
4. Run verification tests:

   python deployment/test_security.py
   cd frontend; npm run build; cd ..

5. Push to your branch: git push origin feature/improved-ocr-parser
6. Open a Pull Request on GitHub.

This project is licensed under the MIT License — see the LICENSE file for details.

CargoVision AI is an independently maintained, local-first computer vision engineering platform derived from and building upon foundational open-source research:

• GARUD Project: Foundational railway wagon inspection research (low-light-image-deblur) created by Tamal Das (Parzival) and collaborators under the MIT License.
• Ultralytics YOLOv8: Object detection and segmentation architecture (Ultralytics).
• NAFNet (Megvii Research): Non-linear Activation Free network for image restoration (Chen et al., ECCV 2022).
• Zero-DCE: Zero-Reference Deep Curve Estimation for low-light image enhancement (Guo et al., CVPR 2020).
• EasyOCR: Optical character recognition framework by JaidedAI.