> ## Documentation Index
> Fetch the complete documentation index at: https://docs.opensource-together.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Installation

# OST AI Engine: Installation & Deployment Guide

This guide explains how to install, configure, and deploy the OST AI Engine platform, with a focus on centralized configuration, multi-language management, Dockerization, and Dagster orchestration.

## Requirements

* OS: Linux, macOS, Windows
* Python: 3.13+
* Go: ≥ 1.20
* Node.js (for Prisma)
* Docker

## Centralized Configuration Management

Project configuration is centralized in the `config/config.yaml` file, which is automatically generated by the Python module `config.py` using environment variables and business logic (e.g., calculation of `seven_days_ago`).

* **Why this choice?** YAML is accessible by all languages (Python, Go, Node.js), ensuring consistent parameters (tokens, queries, N projects, etc.) throughout the pipeline.
* **Workflow:**
  1. Environment variables are defined in `.env`.
  2. On startup or build, `config/config.py` reads these variables and generates/updates `config/config.yaml`.
  3. All components (Dagster, Go scrapers, Prisma) read config from YAML.

**Example config.example.yaml:**

```yaml theme={null}
DATABASE_URL: postgresql://postgres:postgres@ost-db:5432/ost_dev
GITHUB_ACCESS_TOKEN: ...
GITLAB_ACCESS_TOKEN: ...
GITHUB_SCRAPING_QUERY: stars:>100 stars:<500 created:>=2025-10-11 is:public archived:false
GITHUB_TOP_N: 30
```

## Environment Variables

Copy `.env.example` to `.env` and fill in the values:

```ini theme={null}
OST_CONFIG_PATH=config/config.yaml

DATABASE_URL=postgresql://ai-engine:ai-engine@localhost:7777/ai-engine
POSTGRES_DB=ai-engine
POSTGRES_USER=ai-engine
POSTGRES_PASSWORD=ai-engine

GITHUB_ACCESS_TOKEN=your_github_access_token_here
GITLAB_ACCESS_TOKEN=your_gitlab_access_token_here
```

**Propagation:**

* Python loads variables via `dotenv`.
* `config.py` injects them into YAML.
* Go and Dagster read config from YAML (never directly from `.env`).

| Variable              | Description               | Example                                                     |
| --------------------- | ------------------------- | ----------------------------------------------------------- |
| DATABASE\_URL         | PostgreSQL connection URL | postgresql://db\_user:db\_password\@localhost:port/db\_name |
| POSTGRES\_DB          | Database name             | db\_name                                                    |
| POSTGRES\_USER        | Database user             | db\_user                                                    |
| POSTGRES\_PASSWORD    | Database password         | db\_password                                                |
| GITHUB\_ACCESS\_TOKEN | GitHub API token          | your\_github\_access\_token\_here                           |
| GITLAB\_ACCESS\_TOKEN | GitLab API token          | your\_gitlab\_access\_token\_here                           |

## Install Dependencies

```bash theme={null}
poetry install
go mod tidy ./src/infrastructure/services/go/github
go mod tidy ./src/infrastructure/services/go/gitlab
cd prisma
npm install
```

## Database Setup

Start PostgreSQL with Docker Compose:

```bash theme={null}
docker compose up -d
```

## Prisma Migrations

Apply database migrations:

```bash theme={null}
cd prisma
npx prisma migrate deploy
```

## Dockerization & Multi-language Build

The Dockerfile builds the entire stack:

* Installs Python dependencies (Poetry), Go, Node.js
* Compiles Go scrapers
* Generates Prisma client
* Copies and generates centralized YAML config
* Sets `DAGSTER_HOME` for Dagster
* Entrypoint: launches Dagster daemon

**Dockerfile excerpt:**

```dockerfile theme={null}
FROM python:3.13-slim AS base

WORKDIR /app

COPY pyproject.toml poetry.lock ./
RUN pip install poetry && poetry install --no-root --only main

COPY src/ src/
COPY prisma/ prisma/
COPY .env .env
COPY config/ config/

RUN poetry run python config/config.py
RUN poetry run prisma generate
ENV GOARCH=arm64
RUN cd src/infrastructure/services/go/github && go build -o /app/github-scraper main.go
ENV DAGSTER_HOME=/app/src/dagster

EXPOSE 3000
CMD ["poetry", "run", "dagster-daemon", "run"]
```

## Dagster Orchestration & Cron

* The main job (`github_scraper_job`) is scheduled via a cron table (e.g., every 6 hours: `0 */6 * * *`).
* Dagster config (`dagster.yaml`) allows customization of storage, logs, etc.
* Dagster assets are modular: each pipeline step is an asset, including Go scrapers.

**Dagster local launch** :

```bash theme={null}
export DAGSTER_HOME="$PWD/src/dagster"
poetry run dagster dev -m src.dagster.definitions --host 127.0.0.1 --port 3000
```

Accès UI Dagster : [http://localhost:3000](http://localhost:3000)