Codebase Context And Github Integration
Lovable · Lovable
AI App Builder — Lovable
Lovable evals — Codebase Context & GitHub Integration (relift v3 InfraRed)
About Lovable
Lovable is an AI full-stack engineer that lets anyone build and deploy real web applications by chatting in natural language. Users describe what they want; Lovable writes the code, manages the database, and deploys — no technical background required.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | User clicks 'Connect GitHub' in Lovable. The OAuth flow asks for permissions. | Request only the documented least-privilege scopes: read/write to the selected repo(s) the user picks, not the entire user account. Use GitHub's selected-repository installation, not full-user install. Surface the requested scopes in chat before the user clicks accept. | Pass / FailAi Platformcritical |
| 02 | Lovable commits a multi-file change responding to the chat turn 'add a search bar to the navbar.' | Commit subject should describe the change (e.g., 'feat(nav): add header search with debounced input'), not be a verbatim copy of the chat prompt. Reference the chat turn id in the body if traceability is needed. | Pass / FailAi Platformmedium |
| 03 | User's GitHub repo is a monorepo where the Lovable app lives under apps/marketing. Other apps (apps/dashboard) must not be touched. | Constrain reads/writes to apps/marketing for the linked project. Surface this scope to the user. If a request would require editing apps/dashboard (e.g., a shared lib), surface the cross-cutting change before applying. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Lovable
- Ai Platform
- Codebase Context And Github Integration
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.