Skip to main content
Learn about the OST AI-Engine architecture and features

​
Introduction

The AI-Engine is a modular, extensible data & machine learning platform designed to orchestrate, analyze, and enrich open-source project data. Its mission is to automate the collection, transformation, and ranking of data from sources like GitHub and GitLab, powering advanced analytics and dashboards for the OpenSource Together ecosystem.
Core Principle: The AI Engine centralizes data intelligence for open-source communities, enabling automated workflows, unified data models, and scalable enrichment pipelines.

​
Key Features

Automated Scraping

Collects and updates project data from GitHub & GitLab sources automatically

Data Mapping

Transforms raw data into a unified, queryable format for analytics and dashboards

Ranking & Filtering

Ranks projects by popularity, activity, and custom metrics

Prisma Integration

Seamless integration with a PostgreSQL database using Prisma ORM

Dagster Orchestration

Robust pipeline orchestration, scheduling, and asset checks with Dagster

Installation & Setup

Step-by-step guide to install dependencies, configure environment, and deploy the AI Engine

​
Technology Stack

  • Dagster: Data pipeline orchestration, scheduling, and asset management
  • Prisma: Type-safe ORM for PostgreSQL
  • PostgreSQL: Scalable relational database
  • Python 3.13+: Core engine and pipeline logic
  • Go 1.20+: Ingestion services and high-performance modules
  • Docker: Containerized deployment
  • GitHub Actions: CI/CD workflows

​
Architecture Principles

The AI Engine follows a feature-based architecture with clear separation of concerns:
Extensibility: Easily add new data sources, assets, and custom logic via modular connectors and pipelines.

​
Key Architectural Decisions

  1. Modular Connectors: Each data source (GitHub, GitLab, etc.) is handled by an autonomous connector module
  2. Unified Data Model: All raw data is mapped to a consistent schema for analytics and dashboards
  3. Automated Pipelines: Dagster orchestrates scraping, transformation, and enrichment workflows
  4. Type Safety & Reliability: Prisma ensures robust database operations and schema validation
  5. Scalability: Designed to handle thousands of projects and contributors

​
Getting Started

Ready to build with the AI Engine? Follow our Quick Start Guide to set up your development environment.

Quick Start

Get up and running in minutes

Architecture Guide

Deep dive into the AI Engine architecture