PR-AI: Your GitHub QA Copilot

A task-specific, tool-augmented AI agent that analyzes GitHub PRs to generate markdown-based QA summaries and test scenarios using OpenAI or Claude.

• What It Does
• Setup Guide
  • Project can be cloned/forked from link- https://github.com/Hiddensound/Github_QAAgent
  • 1. Install dependencies
  • 2. Create your .env file
  • 3. Run the agent
• Why This Is an AI Agent (Not Just a Script)
  • Why it’s an agent:
• What LangChain Enables
  • Current Role:
  • Future Potential with LangChain:
• Output Example
• Advanced Usage (Future Roadmap)
• Made for QA Teams

What It Does

• Analyzes PR title, files, and comments
• Extracts context and preview links
• Uses an LLM to generate:
  • QA summary
  • Suggested test scenarios and test cases
• Optionally posts the result to GitHub as a comment

Setup Guide

Project can be cloned/forked from link- https://github.com/Hiddensound/Github_QAAgent

1. Install dependencies

npm install dotenv node-fetch cheerio readline fs path \
            @langchain/core @langchain/openai @langchain/anthropic

2. Create your .env file

# .env
OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key
GITHUB_TOKEN=your-classic-github-token # Must have repo/public_repo scope

# Optional LangSmith Tracing & Evaluation
LANGSMITH_API_KEY=your-langsmith-api-key
LANGCHAIN_TRACING=true
LANGCHAIN_PROJECT=github-qa-agent
LANGCHAIN_CALLBACKS_BACKGROUND=true

Note on GitHub Token: This must be a Classic Personal Access Token (not fine-grained). Go to https://github.com/settings/tokens to generate one. Be sure to enable:

• repo scope for private repositories
• public_repo scope for public repositories This allows the agent to fetch PR metadata, comments, and post summaries as comments.

3. Run the agent

For OpenAI:

node githubAgent_openAI.js

For Claude:

node githubAgent_Anthropic.js

It will prompt you for GitHub PR URLs:

Enter GitHub PR URL(s), comma-separated:

Why This Is an AI Agent (Not Just a Script)

What you’ve built is definitely an AI agent, and here’s why:

Why it’s an agent:

• Goal-oriented: It’s not just running commands — it has a clear purpose: to generate actionable QA output from a PR.
• Context-aware: It understands the differences between PRs and tailors its output.
• AI reasoning: The agent uses LLMs to semantically understand file changes, links, and comments.
• Interactive decision-making: It prompts the user to post results or not — this controlled autonomy is agent-like.
• Modular architecture: With LangChain and LangSmith, it leverages chain components, tool integration, and observability — similar to how more advanced agents operate.

This is what’s called a tool-augmented, task-specific agent — not a general-purpose engine like AutoGPT, but far more dynamic and intelligent than a CLI tool.

What LangChain Enables

LangChain plays a crucial role by providing:

Current Role:

• PromptTemplate: dynamically maps GitHub PR content into a structured input for the LLM.
• Model wrapper: use either ChatOpenAI or ChatAnthropic seamlessly.
• Output parsing: converts model output into usable markdown summaries.
• Modular chain: cleanly pipes prompt -> model -> parser, making your QA logic swappable and reusable.
• LangSmith Tracing: visualize traces and debug prompt-level behavior with transparency.

Future Potential with LangChain:

• RAG (Retrieval-Augmented Generation): Inject company test plans, historic test cases, or JIRA specs into model context.
• Memory: Keep track of which PRs have been analyzed and avoid repeat test case generation.
• Tool use: Extend with tools like Lighthouse, TestRail, or analytics dashboards.
• Multi-agent orchestration: Let different agents handle UI vs API test strategy suggestions.

You’re not just calling an API — you’ve structured a modular, interpretable pipeline of intelligence.

Output Example

Markdown file created per PR:

# QA Summary for https://github.com/example/repo/pull/42

## PR Info

Title: Fix checkout bugs
Files Changed:

- pages/checkout.tsx
- utils/payment.js

## QA Summary

- This PR fixes validation issues on the checkout screen...

### Test Scenario: Checkout Form Validations

- [ ] Enter invalid credit card, expect error
- [ ] Submit empty form, expect required field messages

The markdown (.md) files are created for each execution with results as shown in console and automatically saved to: qa_outputs/<repo>-pr<PR#>-<timestamp>.md. User need not create any directory as code creates one automatically if not exisiting.

If user opts in to post the results (QA Summary) to the Github PR, the agent posts the comment on your behalf without your interaction.

Advanced Usage (Future Roadmap)

• Accept input from CSV files
• Add CLI flags (--model=claude, --post)
• Slack summary push
• RAG support for specs/test case injection
• TestRail integration to suggest regression buckets

Made for QA Teams

Built for improving QA review workflows across multiple teams, especially helpful when:

• Streamlining onboarding processes for new team members
• Reducing time spent on writing test cases while ensuring comprehensive coverage
• Addressing gaps in module-specific knowledge
• Enhancing efficiency and maintaining consistency in QA workflows
• Providing clear visibility to approvers on test cases relevant to the PR under review

Want to contribute or improve it? Fork the repo and go wild.