ISOFRAMES
GitHub

OMNISENSE

Multi-camera perception system for person tracking, human state analysis and sensor fusion

Source on GitHub All systems
Status
Research
Version
1.0.0
Platform
GPU workstation
Source
github.com/Sherin-SEF-AI/OmniSense
License
Not specified
01 · Problem

Builds a single, unified picture of a monitored space from several cameras: where people are and what state they are in (attention, posture, drowsiness).

02 · Architecture

How it is put together.

OMNISENSE processes four USB cameras simultaneously to track people in 3D and analyse their state. It detects people with YOLOv8/v9, tracks them with DeepSORT and cross-camera re-identification, fuses positions via multi-view triangulation and Kalman filtering, and runs MediaPipe face, gaze, head-pose, drowsiness (PERCLOS) and body-pose analysis. Output goes to a PyQt6 GUI, a PostgreSQL/TimescaleDB store and REST, WebSocket, MQTT and ROS2 interfaces; a headless mode is available.

Cameraomnisense/camera/Trackingomnisense/tracking/Human stateomnisense/perception/Sensor fusionomnisense/fusion/Intelligenceomnisense/intelligence/Outputsomnisense/api/synchronised framessynchronised framessynchronised framessynchronised framesmulti-camera tracksmulti-camera tracksface / pose / drowsinessface / pose / drowsinessworld stateworld stateCameraomnisense/camera/Trackingomnisense/tracking/Human stateomnisense/perception/Sensor fusionomnisense/fusion/Intelligenceomnisense/intelligence/Outputsomnisense/api/synchronised framessynchronised framessynchronised framessynchronised framesmulti-camera tracksmulti-camera tracksface / pose / drowsinessface / pose / drowsinessworld stateworld state
FIG. 01Modules and data flows, from the repository layout and documentation.
  1. 1

    Camera

    V4L2 capture, frame synchronisation, calibration

    omnisense/camera/
  2. 2

    Tracking

    YOLO detection and DeepSORT multi-camera tracking

    omnisense/tracking/
  3. 3

    Human state

    Face analysis, gaze, pose, drowsiness

    omnisense/perception/
  4. 4

    Sensor fusion

    3D world-state estimation

    omnisense/fusion/
  5. 5

    Intelligence

    Activity, attention, interaction and scene reasoning

    omnisense/intelligence/
  6. 6

    Outputs

    PyQt6 GUI, database, REST/MQTT/ROS2 APIs

    omnisense/api/
03 · Capabilities

Specification.

Cameras
4 synchronised UVC USB cameras via V4L2; intrinsic and stereo calibration CLI
Tracking
YOLOv8/v9 person detection, DeepSORT, cross-camera ReID (OSNet)
3D fusion
Multi-view triangulation with Kalman filtering into a world state
Human state
468-point face mesh, head pose, gaze/attention, PERCLOS drowsiness, 33-keypoint body pose, posture
Interfaces
PyQt6 GUI or headless; REST + WebSocket, MQTT, ROS2 bridge
Storage
PostgreSQL + TimescaleDB schema for tracks, face states, poses, events (optional)
Hardware
NVIDIA GPU with 6 GB+ VRAM, CUDA 11.8+, 16 GB RAM, Ubuntu 22.04
Runs on
README requires an NVIDIA GPU (6 GB+ VRAM, CUDA 11.8+); requirements.txt uses torch and onnxruntime-gpu
Stack
  • Python
  • PyTorch
  • Ultralytics YOLO
  • MediaPipe
  • torchreid
  • OpenCV
  • filterpy
  • Open3D
  • PyQt6
  • FastAPI
  • SQLAlchemy
  • PostgreSQL
  • paho-mqtt
04 · Related

More systems.

Perception and ADAS · LiveLyne ADAS

Real-time, on-device driver assistance for any Android phone.

Perception and ADAS · LiveDashLab

Train object detectors and visual models on dashcam footage without labelling any of it

Next systemTrinity Phase 2

Autonomous vehicle testing and validation suite using a 3-camera infrastructure setup