​

OST AI Engine: Installation & Deployment Guide

​

Requirements

• OS: Linux, macOS, Windows
• Python: 3.13+
• Go: ≥ 1.20
• Node.js (for Prisma)
• Docker

​

Centralized Configuration Management

• Why this choice? YAML is accessible by all languages (Python, Go, Node.js), ensuring consistent parameters (tokens, queries, N projects, etc.) throughout the pipeline.
• Workflow:
  1. Environment variables are defined in .env.
  2. On startup or build, config/config.py reads these variables and generates/updates config/config.yaml.
  3. All components (Dagster, Go scrapers, Prisma) read config from YAML.

DATABASE_URL: postgresql://postgres:postgres@ost-db:5432/ost_dev
GITHUB_ACCESS_TOKEN: ...
GITLAB_ACCESS_TOKEN: ...
GITHUB_SCRAPING_QUERY: stars:>100 stars:<500 created:>=2025-10-11 is:public archived:false
GITHUB_TOP_N: 30

​

Environment Variables

OST_CONFIG_PATH=config/config.yaml

DATABASE_URL=postgresql://ai-engine:ai-engine@localhost:7777/ai-engine
POSTGRES_DB=ai-engine
POSTGRES_USER=ai-engine
POSTGRES_PASSWORD=ai-engine

GITHUB_ACCESS_TOKEN=your_github_access_token_here
GITLAB_ACCESS_TOKEN=your_gitlab_access_token_here

• Python loads variables via dotenv.
• config.py injects them into YAML.
• Go and Dagster read config from YAML (never directly from .env).

​

Install Dependencies

poetry install
go mod tidy ./src/infrastructure/services/go/github
go mod tidy ./src/infrastructure/services/go/gitlab
cd prisma
npm install

​

Database Setup

docker compose up -d

​

Prisma Migrations

cd prisma
npx prisma migrate deploy

​

Dockerization & Multi-language Build

• Installs Python dependencies (Poetry), Go, Node.js
• Compiles Go scrapers
• Generates Prisma client
• Copies and generates centralized YAML config
• Sets DAGSTER_HOME for Dagster
• Entrypoint: launches Dagster daemon

FROM python:3.13-slim AS base

WORKDIR /app

COPY pyproject.toml poetry.lock ./
RUN pip install poetry && poetry install --no-root --only main

COPY src/ src/
COPY prisma/ prisma/
COPY .env .env
COPY config/ config/

RUN poetry run python config/config.py
RUN poetry run prisma generate
ENV GOARCH=arm64
RUN cd src/infrastructure/services/go/github && go build -o /app/github-scraper main.go
ENV DAGSTER_HOME=/app/src/dagster

EXPOSE 3000
CMD ["poetry", "run", "dagster-daemon", "run"]

​

Dagster Orchestration & Cron

• The main job (github_scraper_job) is scheduled via a cron table (e.g., every 6 hours: 0 */6 * * *).
• Dagster config (dagster.yaml) allows customization of storage, logs, etc.
• Dagster assets are modular: each pipeline step is an asset, including Go scrapers.

export DAGSTER_HOME="$PWD/src/dagster"
poetry run dagster dev -m src.dagster.definitions --host 127.0.0.1 --port 3000