Initial commit - 611 cybersecurity skills across all subdomains

This commit is contained in:
mukul975
2026-02-25 10:47:44 +01:00
commit 22a7ab1462
1765 changed files with 280648 additions and 0 deletions
@@ -0,0 +1,318 @@
---
name: integrating-sast-into-github-actions-pipeline
description: >
This skill covers integrating Static Application Security Testing (SAST) tools—CodeQL
and Semgrep—into GitHub Actions CI/CD pipelines. It addresses configuring automated code
scanning on pull requests and pushes, tuning rules to reduce false positives, uploading
SARIF results to GitHub Advanced Security, and establishing quality gates that block merges
when high-severity vulnerabilities are detected.
domain: cybersecurity
subdomain: devsecops
tags: [devsecops, cicd, sast, codeql, semgrep, secure-sdlc]
version: 1.0.0
author: mahipal
license: MIT
---
# Integrating SAST into GitHub Actions Pipeline
## When to Use
- When development teams need automated code-level vulnerability detection on every pull request
- When security teams require consistent SAST enforcement across all repositories in an organization
- When migrating from manual or periodic security reviews to continuous security testing
- When compliance frameworks (SOC 2, PCI DSS, NIST SSDF) require evidence of automated code analysis
- When multiple languages coexist in a monorepo and need unified scanning under one workflow
**Do not use** for runtime vulnerability detection (use DAST instead), for scanning third-party dependencies (use SCA tools like Snyk), or for infrastructure-as-code scanning (use Checkov or tfsec).
## Prerequisites
- GitHub repository with GitHub Actions enabled
- GitHub Advanced Security license (required for CodeQL on private repos; free for public repos)
- Semgrep account for managed rules and Semgrep App dashboard (free tier available)
- Repository code in a supported language: Python, JavaScript/TypeScript, Java, C/C++, C#, Go, Ruby, Swift, Kotlin
## Workflow
### Step 1: Configure CodeQL Analysis Workflow
Create a CodeQL workflow that runs on pull requests and on a weekly schedule to catch vulnerabilities in existing code.
```yaml
# .github/workflows/codeql-analysis.yml
name: "CodeQL Analysis"
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
schedule:
- cron: '30 2 * * 1' # Weekly Monday 2:30 AM
jobs:
analyze:
name: Analyze (${{ matrix.language }})
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: ['javascript', 'python']
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
queries: security-extended,security-and-quality
- name: Autobuild
uses: github/codeql-action/autobuild@v3
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{ matrix.language }}"
```
### Step 2: Add Semgrep Scanning for Custom Rules
Semgrep complements CodeQL with faster scans and support for custom pattern-based rules. Configure it to upload SARIF results to the same GitHub Security tab.
```yaml
# .github/workflows/semgrep.yml
name: "Semgrep SAST Scan"
on:
pull_request:
branches: [main, develop]
push:
branches: [main]
jobs:
semgrep:
name: Semgrep Scan
runs-on: ubuntu-latest
permissions:
security-events: write
contents: read
container:
image: semgrep/semgrep:latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Run Semgrep
run: |
semgrep ci \
--config auto \
--config p/owasp-top-ten \
--config p/cwe-top-25 \
--sarif --output semgrep-results.sarif \
--severity ERROR \
--error
env:
SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}
- name: Upload SARIF
if: always()
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: semgrep-results.sarif
category: semgrep
```
### Step 3: Create Custom Semgrep Rules for Organization Patterns
Write organization-specific rules to catch patterns unique to your codebase, such as deprecated internal APIs or insecure configuration patterns.
```yaml
# .semgrep/custom-rules.yml
rules:
- id: hardcoded-database-url
patterns:
- pattern: |
$DB_URL = "...$PROTO://...:...$PASS@..."
message: |
Hardcoded database connection string with credentials detected.
Use environment variables or a secrets manager instead.
languages: [python, javascript, typescript]
severity: ERROR
metadata:
cwe: "CWE-798: Use of Hard-coded Credentials"
owasp: "A07:2021 - Identification and Authentication Failures"
- id: unsafe-deserialization
patterns:
- pattern-either:
- pattern: pickle.loads(...)
- pattern: yaml.load(..., Loader=yaml.Loader)
- pattern: yaml.load(..., Loader=yaml.FullLoader)
message: |
Unsafe deserialization detected. Use safe alternatives to prevent
remote code execution vulnerabilities.
languages: [python]
severity: ERROR
metadata:
cwe: "CWE-502: Deserialization of Untrusted Data"
- id: missing-csrf-protection
patterns:
- pattern: |
@app.route("...", methods=["POST"])
def $FUNC(...):
...
- pattern-not-inside: |
@csrf.exempt
...
message: "POST endpoint may lack CSRF protection."
languages: [python]
severity: WARNING
```
### Step 4: Establish Quality Gates with Branch Protection
Configure branch protection rules that require SAST checks to pass before merging, preventing vulnerable code from reaching production branches.
```bash
# Use GitHub CLI to set branch protection requiring SAST checks
gh api repos/{owner}/{repo}/branches/main/protection \
--method PUT \
--field required_status_checks='{"strict":true,"contexts":["Analyze (javascript)","Analyze (python)","Semgrep Scan"]}' \
--field enforce_admins=true \
--field required_pull_request_reviews='{"required_approving_review_count":1}'
```
### Step 5: Tune and Suppress False Positives
Manage false positives through CodeQL query filters and Semgrep nosemgrep annotations to maintain developer trust in scan results.
```yaml
# codeql-config.yml - Custom CodeQL configuration
name: "Custom CodeQL Config"
queries:
- uses: security-extended
- uses: security-and-quality
- excludes:
id: js/unused-local-variable
paths-ignore:
- '**/test/**'
- '**/tests/**'
- '**/vendor/**'
- '**/node_modules/**'
- '**/*.test.js'
- '**/*.spec.py'
```
```python
# Example: Suppressing a known false positive in Semgrep
import subprocess
def run_safe_command(cmd_list):
# nosemgrep: python.lang.security.audit.dangerous-subprocess-use
result = subprocess.run(cmd_list, capture_output=True, text=True, shell=False)
return result.stdout
```
### Step 6: Aggregate and Report Findings
Use the GitHub Security Overview dashboard and configure notifications for security alerts across repositories.
```bash
# Query SARIF results via GitHub API for reporting
gh api repos/{owner}/{repo}/code-scanning/alerts \
--jq '.[] | select(.state=="open") | {rule: .rule.id, severity: .rule.security_severity_level, file: .most_recent_instance.location.path, line: .most_recent_instance.location.start_line}'
# Count open alerts by severity
gh api repos/{owner}/{repo}/code-scanning/alerts \
--jq '[.[] | select(.state=="open")] | group_by(.rule.security_severity_level) | map({severity: .[0].rule.security_severity_level, count: length})'
```
## Key Concepts
| Term | Definition |
|------|------------|
| SAST | Static Application Security Testing — analyzes source code without executing it to find security vulnerabilities |
| SARIF | Static Analysis Results Interchange Format — standardized JSON format for expressing results from static analysis tools |
| CodeQL | GitHub's semantic code analysis engine that treats code as data and queries it for vulnerability patterns |
| Semgrep | Lightweight static analysis tool using pattern matching to find bugs and security issues across many languages |
| Security Extended | CodeQL query suite that includes additional security queries beyond the default set for deeper analysis |
| Quality Gate | Automated checkpoint that blocks code from progressing through the pipeline unless security criteria are met |
| False Positive | A scan finding that incorrectly identifies secure code as vulnerable, requiring suppression or tuning |
## Tools & Systems
- **CodeQL**: GitHub's semantic code analysis engine with deep dataflow and taint tracking analysis
- **Semgrep**: Fast, lightweight pattern-matching SAST tool with 3000+ community rules and custom rule support
- **GitHub Advanced Security**: Platform providing code scanning, secret scanning, and dependency review in GitHub
- **SARIF Viewer**: VS Code extension for reviewing SARIF results locally during development
- **GitHub Security Overview**: Organization-level dashboard aggregating security alerts across all repositories
## Common Scenarios
### Scenario: Monorepo with Multiple Languages Needs Unified SAST
**Context**: A platform team manages a monorepo containing Python microservices, TypeScript frontends, and Go infrastructure tools. Security reviews happen manually every quarter, missing vulnerabilities between reviews.
**Approach**:
1. Configure CodeQL with a matrix strategy covering Python, JavaScript, and Go languages
2. Add Semgrep with `--config auto` to detect language automatically and apply relevant rulesets
3. Create path-based triggers so only changed language directories trigger their respective scans
4. Upload all SARIF results to GitHub Security tab with unique categories per tool and language
5. Set branch protection requiring all SAST jobs to pass before merge
6. Schedule weekly full-repository scans to catch issues in unchanged code from newly published CVE patterns
**Pitfalls**: Setting CodeQL to analyze all languages on every PR increases CI time significantly. Use path filters to trigger only relevant language scans. Semgrep's `--config auto` may enable rules that conflict with CodeQL findings, creating duplicate alerts.
### Scenario: Reducing Alert Fatigue from High False Positive Rate
**Context**: After enabling SAST, developers ignore findings because 40% are false positives, undermining the security program.
**Approach**:
1. Export all current alerts and categorize them as true positive, false positive, or informational
2. Create a custom CodeQL config excluding noisy query IDs that produce the most false positives
3. Write `.semgrepignore` patterns for test files, generated code, and vendored dependencies
4. Establish a weekly triage meeting where security and development leads review new rule additions
5. Track false positive rate as a metric and target below 15% for developer trust
**Pitfalls**: Over-suppressing rules to reduce noise can create blind spots. Always validate suppressions against the OWASP Top 10 and CWE Top 25 to ensure critical vulnerability classes remain covered.
## Output Format
```
SAST Pipeline Scan Report
==========================
Repository: org/web-application
Branch: feature/user-auth-refactor
Scan Date: 2026-02-23
Commit: a1b2c3d4
CodeQL Results:
Language Queries Run Findings Critical High Medium
javascript 312 4 1 2 1
python 287 2 0 1 1
Semgrep Results:
Ruleset Rules Matched Findings Errors Warnings
auto 1,847 3 1 2
owasp-top-ten 186 2 1 1
custom-rules 12 1 0 1
QUALITY GATE: FAILED
Blocking findings: 2 Critical/High severity issues
- [CRITICAL] CWE-89: SQL Injection in src/api/users.py:47
- [HIGH] CWE-79: Cross-site Scripting in src/components/Search.tsx:123
Action Required: Fix blocking findings before merge is permitted.
```
@@ -0,0 +1,216 @@
# SAST Pipeline Configuration Templates
## GitHub Actions: Combined CodeQL + Semgrep Workflow
```yaml
# .github/workflows/sast-pipeline.yml
name: "SAST Security Pipeline"
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
schedule:
- cron: '0 3 * * 1'
concurrency:
group: sast-${{ github.ref }}
cancel-in-progress: true
jobs:
# ─────────────── CodeQL Analysis ───────────────
codeql:
name: CodeQL (${{ matrix.language }})
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: ['javascript', 'python']
steps:
- uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
queries: security-extended
config-file: .github/codeql/codeql-config.yml
- name: Autobuild
uses: github/codeql-action/autobuild@v3
- name: Perform Analysis
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{ matrix.language }}"
# ─────────────── Semgrep Scan ───────────────
semgrep:
name: Semgrep Scan
runs-on: ubuntu-latest
permissions:
security-events: write
contents: read
container:
image: semgrep/semgrep:latest
steps:
- uses: actions/checkout@v4
- name: Run Semgrep
run: |
semgrep ci \
--config auto \
--config p/owasp-top-ten \
--config p/cwe-top-25 \
--config .semgrep/ \
--sarif --output semgrep.sarif \
--severity ERROR \
--error
env:
SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}
- name: Upload SARIF
if: always()
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: semgrep.sarif
category: semgrep
# ─────────────── Quality Gate ───────────────
security-gate:
name: Security Quality Gate
needs: [codeql, semgrep]
runs-on: ubuntu-latest
if: always()
steps:
- name: Check SAST Results
run: |
if [ "${{ needs.codeql.result }}" == "failure" ] || [ "${{ needs.semgrep.result }}" == "failure" ]; then
echo "::error::SAST security gate failed. Review findings in the Security tab."
exit 1
fi
echo "Security gate passed."
```
## CodeQL Custom Configuration
```yaml
# .github/codeql/codeql-config.yml
name: "Organization CodeQL Config"
queries:
- uses: security-extended
- uses: security-and-quality
paths-ignore:
- '**/test/**'
- '**/tests/**'
- '**/spec/**'
- '**/vendor/**'
- '**/node_modules/**'
- '**/__mocks__/**'
- '**/*.test.js'
- '**/*.test.ts'
- '**/*.spec.py'
- '**/migrations/**'
query-filters:
- exclude:
id: js/unused-local-variable
- exclude:
id: py/unused-import
```
## Semgrep Ignore File
```
# .semgrepignore
# Test files
*_test.go
*_test.py
*.test.js
*.test.ts
*.spec.js
*.spec.ts
test/
tests/
__tests__/
spec/
# Generated code
*_generated.go
*.pb.go
**/generated/**
# Vendored dependencies
vendor/
node_modules/
third_party/
# Build artifacts
dist/
build/
out/
```
## Branch Protection Configuration (Terraform)
```hcl
# branch-protection.tf
resource "github_branch_protection" "main" {
repository_id = github_repository.app.node_id
pattern = "main"
required_status_checks {
strict = true
contexts = [
"CodeQL (javascript)",
"CodeQL (python)",
"Semgrep Scan",
"Security Quality Gate"
]
}
required_pull_request_reviews {
required_approving_review_count = 1
dismiss_stale_reviews = true
}
enforce_admins = true
allows_deletions = false
allows_force_pushes = false
}
```
## SARIF Report Aggregation Script
```bash
#!/bin/bash
# aggregate-sarif.sh - Merge multiple SARIF files for unified upload
set -euo pipefail
OUTPUT="merged-results.sarif"
SARIF_FILES=($(find . -name "*.sarif" -type f))
if [ ${#SARIF_FILES[@]} -eq 0 ]; then
echo "No SARIF files found"
exit 0
fi
# Use jq to merge SARIF runs
jq -s '{
"$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json",
"version": "2.1.0",
"runs": [.[].runs[]]
}' "${SARIF_FILES[@]}" > "$OUTPUT"
echo "Merged ${#SARIF_FILES[@]} SARIF files into $OUTPUT"
TOTAL=$(jq '[.runs[].results | length] | add' "$OUTPUT")
echo "Total findings: $TOTAL"
```
@@ -0,0 +1,65 @@
# Standards Reference: SAST in GitHub Actions
## OWASP SAMM - Verification: Security Testing
### Maturity Level 1
- Perform automated SAST scanning with default rulesets on all application code
- Results are visible to development teams through IDE or CI/CD integration
### Maturity Level 2
- Customize SAST rules to reduce false positives below 20%
- Track and triage all findings with defined SLAs per severity
- Integrate SAST results into a centralized vulnerability management system
### Maturity Level 3
- Correlate SAST findings with DAST and SCA results for comprehensive coverage
- Measure and improve detection accuracy through benchmarking against known vulnerabilities
- Custom rules cover organization-specific vulnerability patterns and deprecated APIs
## NIST SSDF (SP 800-218) - Produce Well-Secured Software
### PW.7: Review and Analyze Code
- PW.7.1: Determine whether SAST tools should be used and select appropriate tools
- PW.7.2: Use SAST tools to analyze source code and identify vulnerabilities
- Configure tools to analyze code for compliance with secure coding standards
### PW.8: Test Executable Code
- Integration of SAST into CI/CD ensures code is tested before deployment
- Findings are tracked and remediated according to organizational policy
## CIS Software Supply Chain Security Guide
### Source Code (SC) Controls
- SC-2: Enforce branch protection requiring SAST checks to pass
- SC-3: Require code review in addition to automated scanning
- SC-4: Automate security testing in the build pipeline
### Build (BD) Controls
- BD-1: Define and enforce security requirements for build processes
- BD-2: Integrate multiple security testing tools for defense in depth
## OWASP Top 10 Coverage Matrix
| OWASP Category | CodeQL | Semgrep | Combined |
|----------------|--------|---------|----------|
| A01: Broken Access Control | Partial | Yes | Yes |
| A02: Cryptographic Failures | Yes | Yes | Yes |
| A03: Injection | Yes | Yes | Yes |
| A04: Insecure Design | No | Partial | Partial |
| A05: Security Misconfiguration | Partial | Yes | Yes |
| A06: Vulnerable Components | No | No | No (Use SCA) |
| A07: Auth Failures | Yes | Yes | Yes |
| A08: Software Integrity | No | Partial | Partial |
| A09: Logging Failures | Partial | Yes | Yes |
| A10: SSRF | Yes | Yes | Yes |
## PCI DSS v4.0 Mapping
- Requirement 6.2.4: Software engineering techniques or automated methods prevent or mitigate common software attacks
- Requirement 6.3.2: An inventory of bespoke and custom software and third-party software components is maintained
- Requirement 6.5.4: SAST tools are run as part of the software development lifecycle
## SOC 2 Trust Service Criteria
- CC7.1: Deploy detection and monitoring mechanisms for anomalies indicative of actual or attempted attacks
- CC8.1: Authorize, design, develop or acquire, configure, document, test, approve, and implement changes to infrastructure, data, software, and procedures
@@ -0,0 +1,137 @@
# Workflow Reference: SAST in GitHub Actions Pipeline
## End-to-End SAST Integration Workflow
```
Developer Push/PR
┌──────────────────┐
│ GitHub Actions │
│ Trigger │
└──────┬───────────┘
├──────────────────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ CodeQL Init │ │ Semgrep CI │
│ + Autobuild │ │ + Custom │
│ + Analyze │ │ Rules │
└──────┬───────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ SARIF Upload │ │ SARIF Upload │
│ (CodeQL) │ │ (Semgrep) │
└──────┬───────┘ └──────┬───────┘
│ │
└──────────┬─────────┘
┌──────────────────┐
│ GitHub Security │
│ Tab / Dashboard │
└──────┬───────────┘
┌──────────────────┐
│ Branch Protection│
│ Quality Gate │
└──────┬───────────┘
┌─────────┴──────────┐
▼ ▼
PASS: Merge FAIL: Block
Permitted + Notify Dev
```
## CodeQL Analysis Deep Dive
### Database Creation Phase
1. CodeQL extracts source code into a relational database
2. For compiled languages (Java, C++, C#, Go), the build process is intercepted
3. For interpreted languages (Python, JavaScript, Ruby), source files are parsed directly
4. The database captures the full AST, data flow, and control flow of the program
### Query Execution Phase
1. Security queries analyze the database for known vulnerability patterns
2. Taint tracking follows data from untrusted sources to dangerous sinks
3. Dataflow analysis tracks variable assignments across method boundaries
4. Results are deduplicated and ranked by confidence and severity
### Query Suites
- **default**: Core security queries with high precision and low false positive rate
- **security-extended**: Additional queries covering more vulnerability types
- **security-and-quality**: All security queries plus code quality checks
## Semgrep Rule Authoring Process
### Rule Development Lifecycle
1. Identify a vulnerability pattern from a recent security incident or code review
2. Write the pattern using Semgrep syntax with `pattern`, `pattern-either`, `pattern-not`
3. Test the rule against known vulnerable and safe code samples
4. Add metadata: CWE ID, OWASP category, severity, remediation guidance
5. Deploy via `.semgrep/` directory or Semgrep App registry
6. Monitor false positive rate and refine patterns
### Pattern Operators Reference
| Operator | Purpose |
|----------|---------|
| `pattern` | Match a single code pattern |
| `pattern-either` | Match any of multiple patterns (OR) |
| `pattern-not` | Exclude specific patterns from matches |
| `pattern-inside` | Match only within a containing pattern |
| `pattern-not-inside` | Exclude matches within a containing pattern |
| `metavariable-regex` | Constrain metavariable values with regex |
| `metavariable-comparison` | Compare metavariable values numerically |
## SARIF Processing Pipeline
### SARIF Structure
```json
{
"$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json",
"version": "2.1.0",
"runs": [
{
"tool": {
"driver": {
"name": "Semgrep",
"rules": []
}
},
"results": [
{
"ruleId": "hardcoded-database-url",
"level": "error",
"message": { "text": "..." },
"locations": [
{
"physicalLocation": {
"artifactLocation": { "uri": "src/config.py" },
"region": { "startLine": 42 }
}
}
]
}
]
}
]
}
```
## Triage and Remediation Workflow
### Severity-Based SLA
| Severity | Triage SLA | Remediation SLA | Escalation |
|----------|-----------|-----------------|------------|
| Critical | 1 business day | 3 business days | Security Lead + VP Eng |
| High | 3 business days | 10 business days | Security Lead |
| Medium | 5 business days | 30 business days | Team Lead |
| Low | 10 business days | 90 business days | Backlog |
### Finding States
1. **Open**: New finding not yet reviewed
2. **Confirmed**: Finding validated as true positive
3. **False Positive**: Finding dismissed with justification
4. **Fixed**: Remediation committed and verified by rescan
5. **Won't Fix**: Accepted risk with documented justification and risk owner
@@ -0,0 +1,380 @@
#!/usr/bin/env python3
"""
SAST Pipeline Orchestration Script
Runs CodeQL and Semgrep scans, aggregates SARIF results, evaluates quality gates,
and produces a consolidated report. Designed to be invoked from GitHub Actions
or any CI/CD platform.
Usage:
python process.py --repo-path /path/to/repo --output report.json
python process.py --repo-path . --severity-threshold high --fail-on-findings
"""
import argparse
import json
import os
import subprocess
import sys
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
@dataclass
class ScanFinding:
rule_id: str
severity: str
message: str
file_path: str
start_line: int
end_line: int
tool: str
cwe: str = ""
owasp: str = ""
fingerprint: str = ""
@dataclass
class ScanResult:
tool: str
findings: list = field(default_factory=list)
rules_evaluated: int = 0
scan_duration_seconds: float = 0.0
exit_code: int = 0
error_message: str = ""
SEVERITY_ORDER = {"critical": 0, "high": 1, "medium": 2, "low": 3, "note": 4, "none": 5}
def run_semgrep(repo_path: str, config: str = "auto", extra_configs: Optional[list] = None) -> ScanResult:
"""Run Semgrep scan and return structured results."""
result = ScanResult(tool="semgrep")
sarif_output = os.path.join(repo_path, "semgrep-results.sarif")
cmd = [
"semgrep", "ci",
"--config", config,
"--sarif",
"--output", sarif_output,
"--json",
"--quiet"
]
if extra_configs:
for cfg in extra_configs:
cmd.extend(["--config", cfg])
start_time = datetime.now(timezone.utc)
try:
proc = subprocess.run(
cmd,
cwd=repo_path,
capture_output=True,
text=True,
timeout=600
)
result.exit_code = proc.returncode
if proc.returncode not in (0, 1):
result.error_message = proc.stderr[:500]
return result
except subprocess.TimeoutExpired:
result.error_message = "Semgrep scan timed out after 600 seconds"
result.exit_code = -1
return result
except FileNotFoundError:
result.error_message = "semgrep binary not found. Install with: pip install semgrep"
result.exit_code = -1
return result
result.scan_duration_seconds = (datetime.now(timezone.utc) - start_time).total_seconds()
if os.path.exists(sarif_output):
result.findings = parse_sarif(sarif_output, "semgrep")
with open(sarif_output, "r") as f:
sarif_data = json.load(f)
for run in sarif_data.get("runs", []):
result.rules_evaluated = len(run.get("tool", {}).get("driver", {}).get("rules", []))
return result
def run_codeql_query(repo_path: str, language: str, database_path: str) -> ScanResult:
"""Run CodeQL analysis on a pre-created database and return structured results."""
result = ScanResult(tool=f"codeql-{language}")
sarif_output = os.path.join(repo_path, f"codeql-{language}-results.sarif")
cmd = [
"codeql", "database", "analyze",
database_path,
f"codeql/{language}-queries:codeql-suites/{language}-security-extended.qls",
"--format=sarifv2.1.0",
f"--output={sarif_output}",
"--threads=0"
]
start_time = datetime.now(timezone.utc)
try:
proc = subprocess.run(
cmd,
cwd=repo_path,
capture_output=True,
text=True,
timeout=1200
)
result.exit_code = proc.returncode
if proc.returncode != 0:
result.error_message = proc.stderr[:500]
except subprocess.TimeoutExpired:
result.error_message = "CodeQL analysis timed out after 1200 seconds"
result.exit_code = -1
return result
except FileNotFoundError:
result.error_message = "codeql binary not found. Install from https://github.com/github/codeql-cli-binaries"
result.exit_code = -1
return result
result.scan_duration_seconds = (datetime.now(timezone.utc) - start_time).total_seconds()
if os.path.exists(sarif_output):
result.findings = parse_sarif(sarif_output, f"codeql-{language}")
with open(sarif_output, "r") as f:
sarif_data = json.load(f)
for run in sarif_data.get("runs", []):
result.rules_evaluated = len(run.get("tool", {}).get("driver", {}).get("rules", []))
return result
def parse_sarif(sarif_path: str, tool_name: str) -> list:
"""Parse a SARIF file and extract findings as ScanFinding objects."""
findings = []
with open(sarif_path, "r") as f:
sarif_data = json.load(f)
for run in sarif_data.get("runs", []):
rules_map = {}
for rule in run.get("tool", {}).get("driver", {}).get("rules", []):
rule_id = rule.get("id", "")
properties = rule.get("properties", {})
cwe_tags = [t for t in properties.get("tags", []) if t.startswith("CWE")]
owasp_tags = [t for t in properties.get("tags", []) if "owasp" in t.lower()]
rules_map[rule_id] = {
"cwe": cwe_tags[0] if cwe_tags else "",
"owasp": owasp_tags[0] if owasp_tags else "",
"severity": rule.get("defaultConfiguration", {}).get("level", "warning")
}
for result in run.get("results", []):
rule_id = result.get("ruleId", "unknown")
rule_info = rules_map.get(rule_id, {})
level = result.get("level", rule_info.get("severity", "warning"))
severity_map = {"error": "high", "warning": "medium", "note": "low", "none": "none"}
severity = severity_map.get(level, "medium")
security_severity = None
for rule in run.get("tool", {}).get("driver", {}).get("rules", []):
if rule.get("id") == rule_id:
security_severity = rule.get("properties", {}).get("security-severity")
break
if security_severity:
score = float(security_severity)
if score >= 9.0:
severity = "critical"
elif score >= 7.0:
severity = "high"
elif score >= 4.0:
severity = "medium"
else:
severity = "low"
locations = result.get("locations", [{}])
physical = locations[0].get("physicalLocation", {}) if locations else {}
artifact = physical.get("artifactLocation", {})
region = physical.get("region", {})
findings.append(ScanFinding(
rule_id=rule_id,
severity=severity,
message=result.get("message", {}).get("text", ""),
file_path=artifact.get("uri", "unknown"),
start_line=region.get("startLine", 0),
end_line=region.get("endLine", region.get("startLine", 0)),
tool=tool_name,
cwe=rule_info.get("cwe", ""),
owasp=rule_info.get("owasp", ""),
fingerprint=str(result.get("fingerprints", {}).get("primaryLocationLineHash", ""))
))
return findings
def evaluate_quality_gate(findings: list, severity_threshold: str) -> dict:
"""Evaluate quality gate based on finding severities."""
threshold_level = SEVERITY_ORDER.get(severity_threshold.lower(), 1)
blocking_findings = [
f for f in findings
if SEVERITY_ORDER.get(f.severity.lower(), 5) <= threshold_level
]
severity_counts = {}
for f in findings:
sev = f.severity.lower()
severity_counts[sev] = severity_counts.get(sev, 0) + 1
return {
"passed": len(blocking_findings) == 0,
"threshold": severity_threshold,
"total_findings": len(findings),
"blocking_findings": len(blocking_findings),
"severity_counts": severity_counts,
"blocking_details": [
{
"rule_id": f.rule_id,
"severity": f.severity,
"file": f.file_path,
"line": f.start_line,
"tool": f.tool,
"message": f.message[:200]
}
for f in blocking_findings
]
}
def generate_report(scan_results: list, quality_gate: dict, repo_path: str) -> dict:
"""Generate a consolidated SAST report."""
all_findings = []
for sr in scan_results:
all_findings.extend(sr.findings)
cwe_counts = {}
for f in all_findings:
if f.cwe:
cwe_counts[f.cwe] = cwe_counts.get(f.cwe, 0) + 1
report = {
"report_metadata": {
"generated_at": datetime.now(timezone.utc).isoformat(),
"repository": repo_path,
"report_version": "1.0.0"
},
"scan_summary": [
{
"tool": sr.tool,
"findings_count": len(sr.findings),
"rules_evaluated": sr.rules_evaluated,
"duration_seconds": sr.scan_duration_seconds,
"status": "success" if sr.exit_code in (0, 1) else "error",
"error": sr.error_message
}
for sr in scan_results
],
"quality_gate": quality_gate,
"top_cwes": sorted(cwe_counts.items(), key=lambda x: x[1], reverse=True)[:10],
"findings": [
{
"rule_id": f.rule_id,
"severity": f.severity,
"tool": f.tool,
"file": f.file_path,
"line": f.start_line,
"cwe": f.cwe,
"owasp": f.owasp,
"message": f.message[:300]
}
for f in sorted(all_findings, key=lambda x: SEVERITY_ORDER.get(x.severity.lower(), 5))
]
}
return report
def main():
parser = argparse.ArgumentParser(description="SAST Pipeline Orchestration")
parser.add_argument("--repo-path", required=True, help="Path to the repository to scan")
parser.add_argument("--output", default="sast-report.json", help="Output report file path")
parser.add_argument("--severity-threshold", default="high",
choices=["critical", "high", "medium", "low"],
help="Minimum severity to block pipeline")
parser.add_argument("--fail-on-findings", action="store_true",
help="Exit with non-zero code if quality gate fails")
parser.add_argument("--semgrep-config", default="auto",
help="Semgrep configuration (default: auto)")
parser.add_argument("--semgrep-extra-configs", nargs="*",
help="Additional Semgrep config paths")
parser.add_argument("--skip-semgrep", action="store_true", help="Skip Semgrep scan")
parser.add_argument("--skip-codeql", action="store_true", help="Skip CodeQL scan")
parser.add_argument("--codeql-language", default=None, help="Language for CodeQL analysis")
parser.add_argument("--codeql-db-path", default=None, help="Path to CodeQL database")
parser.add_argument("--sarif-only", nargs="*",
help="Only parse existing SARIF files instead of running scans")
args = parser.parse_args()
repo_path = os.path.abspath(args.repo_path)
scan_results = []
if args.sarif_only:
for sarif_file in args.sarif_only:
tool_name = Path(sarif_file).stem
findings = parse_sarif(sarif_file, tool_name)
sr = ScanResult(tool=tool_name, findings=findings)
scan_results.append(sr)
else:
if not args.skip_semgrep:
print("[*] Running Semgrep scan...")
semgrep_result = run_semgrep(
repo_path,
config=args.semgrep_config,
extra_configs=args.semgrep_extra_configs
)
scan_results.append(semgrep_result)
print(f" Found {len(semgrep_result.findings)} findings in {semgrep_result.scan_duration_seconds:.1f}s")
if semgrep_result.error_message:
print(f" Warning: {semgrep_result.error_message}")
if not args.skip_codeql and args.codeql_language and args.codeql_db_path:
print(f"[*] Running CodeQL analysis for {args.codeql_language}...")
codeql_result = run_codeql_query(repo_path, args.codeql_language, args.codeql_db_path)
scan_results.append(codeql_result)
print(f" Found {len(codeql_result.findings)} findings in {codeql_result.scan_duration_seconds:.1f}s")
all_findings = []
for sr in scan_results:
all_findings.extend(sr.findings)
quality_gate = evaluate_quality_gate(all_findings, args.severity_threshold)
report = generate_report(scan_results, quality_gate, repo_path)
output_path = os.path.abspath(args.output)
with open(output_path, "w") as f:
json.dump(report, f, indent=2)
print(f"\n[*] Report written to {output_path}")
if quality_gate["passed"]:
print(f"[PASS] Quality gate passed. {quality_gate['total_findings']} findings, none blocking.")
else:
print(f"[FAIL] Quality gate failed. {quality_gate['blocking_findings']} blocking findings:")
for detail in quality_gate["blocking_details"]:
print(f" - [{detail['severity'].upper()}] {detail['rule_id']} in {detail['file']}:{detail['line']}")
if args.fail_on_findings and not quality_gate["passed"]:
sys.exit(1)
if __name__ == "__main__":
main()