Skip to main content
The OST AI Engine pipeline orchestrates the scraping, ranking, normalization, and insertion of trending open-source projects into the database. Dagster manages the workflow, ensuring each step is executed reliably and data quality is maintained.

​
Job Scheduling & Definition

Below are visual diagrams showing how the GitHub scraping job is scheduled and defined in Dagster:

Job Schedule: The pipeline is scheduled to run automatically every 6 hours, keeping the database up to date with the latest trending projects and respecting the rate-limiting of the GitHub API.

Job Definition: The job includes four main assets: scraping, ranking, mapping, and database insertion, with data quality checks at each stage.

Assets Overview: Each asset represents a distinct step in the pipeline, from data collection to database insertion, ensuring modularity and maintainability.

​
Asset Checks

Dagster asset checks validate the quality and consistency of data at each pipeline step. Here are the main checks used in the OST AI Engine pipeline: Below are visual examples of asset checks in the Dagster UI:

Checks Overview: The Dagster UI displays the status of each asset check, making it easy to monitor data quality in real time.

Checks Details: Detailed results show which projects passed or failed each check, with error messages for troubleshooting and data validation. All errors and warnings are surfaced in the Dagster UI for rapid debugging.