ISOFRAMES
GitHub

Falcon Autonomy

One desktop application for the whole loop of building a self-driving AI on CARLA

Source on GitHub All systems
Status
Live
Version
2.3.0
Platform
Desktop app
Source
github.com/Sherin-SEF-AI/Falcon-Autonomy
License
GPL-3.0
01 · Problem

Building a self-driving model involves many separate tools for worlds, data, scenarios, training, evaluation and stack testing. Falcon puts the loop of generate, train, test, find failures and generate more exactly there in one application, with each result reproducible from its run.

02 · Architecture

How it is put together.

Falcon Autonomy is a desktop application for the whole loop of building a self-driving AI on the CARLA 0.9.16 simulator. You build or import a world, drive it, script and search scenarios, generate labelled data where models are weak, train and evaluate, and test a full AV stack. Every number is traceable to the run that produced it. Its Loop evaluates a model, finds the weak cells, searches for rare cases there, generates targeted data, retrains and evaluates again on the same held-out set.

Desktop UIfalcon_autonomy/ui/Corefalcon_autonomy/core/CARLA servicefalcon_autonomy/sim/Workersfalcon_autonomy/workers/ROS 2 bridgefalcon_autonomy/ros2/Hardware in the loopfalcon_autonomy/hil/Assistantfalcon_autonomy/assistant/builds a job configbuilds a job confighands the config to CarlaServicehands the config to Car…JSON lines over stdin/stdoutJSON lines over stdin/s…run.json indexed in the cataloguerun.json indexed in the…Desktop UIfalcon_autonomy/ui/Corefalcon_autonomy/core/CARLA servicefalcon_autonomy/sim/Workersfalcon_autonomy/workers/ROS 2 bridgefalcon_autonomy/ros2/Hardware in the loopfalcon_autonomy/hil/Assistantfalcon_autonomy/assistant/builds a job configbuilds a job confighands the config to CarlaServicehands the config to Car…JSON lines over stdin/stdoutJSON lines over stdin/s…run.json indexed in the cataloguerun.json indexed in the…
FIG. 01Modules and data flows, from the repository layout and documentation.
  1. 1

    Desktop UI

    PyQt6 main window, pages, first-run wizard and command palette

    falcon_autonomy/ui/
  2. 2

    Core

    Process-neutral project format and edits, provenance, SQLite catalogue, job queue with locks and memory guard, budgets, labels, drive configs

    falcon_autonomy/core/
  3. 3

    CARLA service

    Finds CARLA, runs Falcon's own server, and turns 'run this on CARLA' into a queued job for the GUI and CLI

    falcon_autonomy/sim/
  4. 4

    Workers

    Processes in CARLA's own Python that drive, record and write run.json, speaking a JSON-lines protocol

    falcon_autonomy/workers/
  5. 5

    ROS 2 bridge

    Closed-loop bridge for testing an AV stack against CARLA

    falcon_autonomy/ros2/
  6. 6

    Hardware in the loop

    Camera and lidar streamed to a Jetson or simulated device, control back, latency per stage

    falcon_autonomy/hil/
  7. 7

    Assistant

    An agent that uses the app through typed tools, with approval cards and undo

    falcon_autonomy/assistant/
03 · Capabilities

Specification.

Simulator and worlds
CARLA 0.9.16 on a Falcon-managed headless server; CARLA towns with weather, OpenStreetMap imports, left-hand traffic and an Indian mixed-traffic profile
Labelled data
Semantic, instance, range, 2D and 3D boxes, occlusion, track ids, lane lines, lidar point labels; exports to COCO, YOLO, KITTI, nuScenes, Waymo-style, BDD lanes, ROS 2 MCAP
Scenarios and rare cases
Scenario studio with eleven built-ins (four for Indian roads); TTC, PET and other criticality measures; Bayesian optimisation, CMA-ES and adaptive-sampling search; OpenSCENARIO import/export
Models and evaluation
ONNX, TorchScript or plug-in detectors and segmenters scored with COCO AP/AR or mIoU and bootstrap intervals; sim-to-real gap against KITTI, nuScenes, BDD100K or COCO
Training
torchvision detector fine-tuning exported to ONNX; Gymnasium environment, behaviour cloning and PPO with a scenario curriculum; world-model and policy training
Stack testing
Closed-loop ROS 2 bridge (Autoware, Ackermann or CARLA control), SOTIF-style safety-case report, log replay of nuScenes, Waymo v2, ROS 2 bags and KITTI raw
Deployment
TensorRT engines built, profiled and accuracy-checked on a Jetson with a release gate; hardware-in-the-loop streaming; CI mode with JUnit/Markdown/HTML outputs
Platform
Linux x86_64 with an NVIDIA GPU, its Vulkan driver and systemd; a CLI for every heavy action
Runs on
PyQt6 desktop application (gui-script falcon-gui) that also needs a local NVIDIA GPU with Vulkan and a CARLA 0.9.16 install
Stack
  • Python
  • PyQt6
  • NumPy
  • OpenCV
  • jsonschema
  • pyqtgraph
  • PyOpenGL
  • CARLA 0.9.16
  • SQLite
  • PyTorch
  • TensorRT
  • ROS 2
  • PyInstaller
  • NVIDIA Jetson AGX Orin (deployment target)
04 · Changelog

Releases.

  1. v2.3.0
    Point release after the 2.0 deployment milestone.
  2. v2.2.0
    Milestones 9 to 13 in one day, ending with TensorRT deployment on the Jetson AGX Orin.
  3. v2.1.0
    Milestones 9 to 13 in one day, ending with TensorRT deployment on the Jetson AGX Orin.
  4. v2.0.0
    Milestones 9 to 13 in one day, ending with TensorRT deployment on the Jetson AGX Orin.
  5. v1.4.0
    Milestones 9 to 13 in one day, ending with TensorRT deployment on the Jetson AGX Orin.
  6. v1.3.0
    Milestones 9 to 13 in one day, ending with TensorRT deployment on the Jetson AGX Orin.
  7. v1.2.0
    Milestones 9 to 13 in one day, ending with TensorRT deployment on the Jetson AGX Orin.
  8. v1.1.0
    Milestones 9 to 13 in one day, ending with TensorRT deployment on the Jetson AGX Orin.
  9. v1.0.0
    Milestones 9 to 13 in one day, ending with TensorRT deployment on the Jetson AGX Orin.
  10. v0.6.0
    Models and the sim-to-real gap, then the training lab.
  11. v0.5.0
    Models and the sim-to-real gap, then the training lab.
05 · Related

More systems.

Simulation · ResearchAutoSim Pro

Autonomous vehicle simulation platform with vehicle physics, AI behaviours and analytics

Next systemLyne ADAS

Real-time, on-device driver assistance for any Android phone.