# Testing Stack Gap Analysis & Tool Recommendations

Analysis of gaps in the current testing-security-stack with recommended tools to fill each gap.

## Current Stack Coverage

### What We Have

| Category | Tool | Coverage |
|----------|------|----------|
| SAST | Semgrep | Source code analysis |
| SAST | SonarQube | Code quality & security |
| DAST | OWASP ZAP | Web app scanning |
| DAST | Nuclei | Template-based scanning |
| Container | Trivy | Image & dependency scanning |
| Secrets | Gitleaks | Secret detection |
| Performance | K6 | Load testing |
| Performance | Locust | Python load testing |
| Performance | Artillery | Scenario-based testing |
| API | Newman | Collection testing |
| API | WireMock | API mocking |
| E2E | Playwright | Browser automation |
| AI Security | CAI | AI-powered testing |
| Reporting | Allure | Test reporting |

### Identified Gaps

## Gap 1: Infrastructure as Code (IaC) Security

**Current State**: No dedicated IaC scanning
**Impact**: Terraform, CloudFormation, Kubernetes manifests not scanned for misconfigurations

### Recommended Tools

| Tool | Best For | Docker Image |
|------|----------|--------------|
| **Checkov** (Recommended) | Comprehensive IaC scanning | `bridgecrew/checkov:latest` |
| KICS | Multi-framework scanning | `checkmarx/kics:latest` |
| Terrascan | Policy-as-code | `accurics/terrascan:latest` |
| tfsec | Terraform-specific | `aquasec/tfsec:latest` |

**Recommendation**: Add **Checkov** - supports 40+ frameworks including Terraform, CloudFormation, Kubernetes, Helm, Dockerfile.

```yaml
# Add to docker-compose.yml
checkov:
  image: bridgecrew/checkov:latest
  volumes:
    - ./code:/code:ro
  command: checkov -d /code --output sarif --output-file /code/checkov.sarif
```

```bash
# Usage
checkov -d . --framework terraform
checkov -d . --framework kubernetes
checkov -d . --framework dockerfile
```

---

## Gap 2: Contract Testing

**Current State**: API testing with Newman, but no consumer-driven contract testing
**Impact**: Breaking API changes not caught before deployment

### Recommended Tools

| Tool | Best For | Docker Image |
|------|----------|--------------|
| **Pact** (Recommended) | Consumer-driven contracts | `pactfoundation/pact-cli:latest` |
| Specmatic | OpenAPI contract testing | N/A (CLI) |
| Dredd | API Blueprint/OpenAPI | `apiaryio/dredd:latest` |

**Recommendation**: Add **Pact** - industry standard for contract testing, excellent broker capabilities.

```yaml
# Add to docker-compose.yml
pact-broker:
  image: pactfoundation/pact-broker:latest
  environment:
    PACT_BROKER_DATABASE_URL: "sqlite:///pact_broker.sqlite3"
    PACT_BROKER_BASIC_AUTH_USERNAME: admin
    PACT_BROKER_BASIC_AUTH_PASSWORD: admin
  ports:
    - "9292:9292"
```

```bash
# Consumer test
pact-mock-service start --port 1234
# Run consumer tests that generate pact files
pact-mock-service stop

# Provider verification
pact-provider-verifier --provider-base-url http://api:8080 --pact-urls ./pacts/*.json
```

---

## Gap 3: Visual Regression Testing

**Current State**: Playwright has basic screenshot comparison
**Impact**: Visual changes not systematically tracked across releases

### Recommended Tools

| Tool | Best For | Type |
|------|----------|------|
| **BackstopJS** (Recommended) | CI/CD visual regression | Open Source |
| Playwright Screenshots | Built-in comparison | Already have |
| reg-suit | Git-based baseline | Open Source |
| Loki | Storybook components | Open Source |

**Recommendation**: Add **BackstopJS** - excellent CI/CD integration, Docker support.

```yaml
# Add to docker-compose.yml
backstopjs:
  image: backstopjs/backstopjs:latest
  volumes:
    - ./backstop:/src
  working_dir: /src
```

```bash
# Initialize
backstop init

# Reference screenshots
backstop reference

# Run comparison
backstop test

# Approve changes
backstop approve
```

---

## Gap 4: Accessibility Testing

**Current State**: No accessibility testing
**Impact**: WCAG compliance issues not detected

### Recommended Tools

| Tool | Best For | Integration |
|------|----------|-------------|
| **axe-core** (Recommended) | Programmatic testing | Playwright/Jest |
| Pa11y | CLI & CI/CD | Standalone |
| Lighthouse | Overall audit | Chrome |
| jest-axe | Unit test style | Jest |

**Recommendation**: Add **axe-core** with Playwright integration.

```javascript
// playwright.config.js addition
const { injectAxe, checkA11y } = require('axe-playwright');

test('@accessibility Homepage audit', async ({ page }) => {
  await page.goto('/');
  await injectAxe(page);
  await checkA11y(page, null, {
    detailedReport: true,
    detailedReportOptions: { html: true }
  });
});
```

```bash
# Add Pa11y for CI
npm install -g pa11y
pa11y http://target:8080 --reporter cli
pa11y http://target:8080 --reporter json > a11y-report.json
```

---

## Gap 5: Chaos Engineering

**Current State**: No resilience testing
**Impact**: System behavior under failure conditions unknown

### Recommended Tools

| Tool | Best For | Environment |
|------|----------|-------------|
| **Toxiproxy** (Recommended) | Network failure simulation | Docker |
| Litmus | Kubernetes chaos | Kubernetes |
| Chaos Mesh | Kubernetes-native | Kubernetes |
| Pumba | Docker chaos | Docker |

**Recommendation**: Add **Toxiproxy** for network chaos.

```yaml
# Add to docker-compose.yml
toxiproxy:
  image: ghcr.io/shopify/toxiproxy:latest
  ports:
    - "8474:8474"   # API
    - "19000-19100:19000-19100"  # Proxies

toxiproxy-cli:
  image: ghcr.io/shopify/toxiproxy:latest
  entrypoint: /toxiproxy-cli
```

```bash
# Create proxy
toxiproxy-cli create myapp -l 0.0.0.0:19000 -u api:8080

# Add latency
toxiproxy-cli toxic add myapp -t latency -a latency=1000

# Add network timeout
toxiproxy-cli toxic add myapp -t timeout -a timeout=5000

# Simulate connection reset
toxiproxy-cli toxic add myapp -t reset_peer -a timeout=1000
```

---

## Gap 6: Fuzzing / Security Fuzzing

**Current State**: No API or input fuzzing
**Impact**: Edge cases and malformed input vulnerabilities not discovered

### Recommended Tools

| Tool | Best For | Type |
|------|----------|------|
| **RESTler** (Recommended) | API fuzzing | Stateful |
| AFL++ | Binary fuzzing | Coverage-guided |
| Jazzer | Java fuzzing | OSS-Fuzz compatible |
| go-fuzz | Go fuzzing | Built-in (Go 1.18+) |

**Recommendation**: Add **RESTler** for API fuzzing.

```yaml
# Add to docker-compose.yml
restler:
  image: mcr.microsoft.com/restlerfuzzer/restler:latest
  volumes:
    - ./restler:/restler
  working_dir: /restler
```

```bash
# Compile from OpenAPI spec
restler compile --api_spec /restler/openapi.json

# Quick test (sanity check)
restler test --grammar_file /restler/Compile/grammar.py

# Fuzz testing
restler fuzz-lean --grammar_file /restler/Compile/grammar.py --time_budget 1h
```

---

## Gap 7: Supply Chain Security

**Current State**: Trivy scans dependencies, but no SBOM generation or artifact signing
**Impact**: No software supply chain attestation or integrity verification

### Recommended Tools

| Tool | Purpose | Integration |
|------|---------|-------------|
| **Syft** (Recommended) | SBOM generation | CLI/Docker |
| **cosign** (Recommended) | Artifact signing | Sigstore |
| Grype | Vulnerability scanning | Pairs with Syft |
| in-toto | Attestations | Build pipeline |

**Recommendation**: Add **Syft** + **cosign** for supply chain security.

```yaml
# Add to docker-compose.yml
syft:
  image: anchore/syft:latest
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - ./sboms:/sboms
```

```bash
# Generate SBOM
syft myapp:latest -o cyclonedx-json > sbom.json
syft myapp:latest -o spdx-json > sbom-spdx.json

# Sign container image
cosign sign --key cosign.key myapp:latest

# Verify signature
cosign verify --key cosign.pub myapp:latest

# Attach SBOM to image
cosign attach sbom --sbom sbom.json myapp:latest
```

---

## Gap 8: Vulnerability Management / Aggregation

**Current State**: Multiple scanners output separate reports
**Impact**: No centralized vulnerability tracking or deduplication

### Recommended Tools

| Tool | Purpose | Integration |
|------|---------|-------------|
| **DefectDojo** (Recommended) | Vulnerability management | All scanners |
| OWASP Dependency-Track | SBOM-based tracking | Trivy/Syft |
| Faraday | Security workspace | Pentesting tools |

**Recommendation**: Add **DefectDojo** for centralized vulnerability management.

```yaml
# Add to docker-compose.yml
defectdojo:
  image: defectdojo/defectdojo-django:latest
  environment:
    DD_DATABASE_URL: "postgresql://defectdojo:defectdojo@defectdojo-db:5432/defectdojo"
    DD_SECRET_KEY: "your-secret-key"
  ports:
    - "8080:8080"
  depends_on:
    - defectdojo-db

defectdojo-db:
  image: postgres:14
  environment:
    POSTGRES_USER: defectdojo
    POSTGRES_PASSWORD: defectdojo
    POSTGRES_DB: defectdojo
```

```bash
# Import scan results
curl -X POST "http://localhost:8080/api/v2/import-scan/" \
  -H "Authorization: Token YOUR_TOKEN" \
  -F "scan_type=Trivy Scan" \
  -F "file=@trivy.json" \
  -F "engagement=1"
```

---

## Gap 9: Runtime Security Monitoring

**Current State**: No runtime security monitoring
**Impact**: Runtime threats not detected

### Recommended Tools

| Tool | Purpose | Environment |
|------|---------|-------------|
| **Falco** (Recommended) | Runtime threat detection | K8s/Docker |
| Sysdig | Deep visibility | K8s/Docker |
| Tracee | eBPF-based detection | Linux |

**Recommendation**: Add **Falco** for runtime security.

```yaml
# Add to docker-compose.yml
falco:
  image: falcosecurity/falco:latest
  privileged: true
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - /dev:/host/dev
    - /proc:/host/proc:ro
    - ./falco-rules:/etc/falco/rules.d:ro
```

---

## Gap 10: Database Testing

**Current State**: No database-specific testing
**Impact**: Schema migrations, data integrity not validated

### Recommended Tools

| Tool | Purpose | Database |
|------|---------|----------|
| **Flyway** (Recommended) | Migration management | All |
| pgTAP | Unit testing | PostgreSQL |
| tSQLt | Unit testing | SQL Server |
| Great Expectations | Data validation | All |

**Recommendation**: Add **Flyway** for migration testing.

```yaml
# Add to docker-compose.yml
flyway:
  image: flyway/flyway:latest
  volumes:
    - ./sql:/flyway/sql
  command: migrate
  environment:
    FLYWAY_URL: jdbc:postgresql://db:5432/myapp
    FLYWAY_USER: postgres
    FLYWAY_PASSWORD: password
```

---

## Priority Ranking

### Tier 1 - Critical Gaps (Add Immediately)

| Priority | Gap | Recommended Tool |
|----------|-----|------------------|
| 1 | IaC Security | Checkov |
| 2 | Vulnerability Management | DefectDojo |
| 3 | Supply Chain Security | Syft + cosign |

### Tier 2 - Important Gaps (Add Soon)

| Priority | Gap | Recommended Tool |
|----------|-----|------------------|
| 4 | Contract Testing | Pact |
| 5 | Accessibility Testing | axe-core |
| 6 | Chaos Engineering | Toxiproxy |

### Tier 3 - Nice to Have (Add as Needed)

| Priority | Gap | Recommended Tool |
|----------|-----|------------------|
| 7 | Fuzzing | RESTler |
| 8 | Visual Regression | BackstopJS |
| 9 | Runtime Security | Falco |
| 10 | Database Testing | Flyway |

---

## Implementation Roadmap

### Phase 1: Security Hardening (Week 1-2)
- Add Checkov for IaC scanning
- Add DefectDojo for vulnerability aggregation
- Add Syft/cosign for supply chain security

### Phase 2: Testing Enhancement (Week 3-4)
- Add Pact for contract testing
- Add axe-core for accessibility
- Add Toxiproxy for chaos engineering

### Phase 3: Advanced Testing (Week 5-6)
- Add RESTler for fuzzing
- Add BackstopJS for visual regression
- Configure Falco for runtime security

### Phase 4: Data & Database (Week 7-8)
- Add Flyway for migration testing
- Add Great Expectations for data validation
- Integrate all tools into CI/CD pipeline

---

## Updated Tool Decision Tree

```
What are you testing?
│
├── Source Code Security
│   ├── Static Analysis → Semgrep
│   ├── Code Quality → SonarQube
│   ├── Secrets → Gitleaks
│   └── IaC Security → Checkov [NEW]
│
├── Dependencies & Supply Chain
│   ├── Vulnerabilities → Trivy
│   ├── SBOM Generation → Syft [NEW]
│   ├── Artifact Signing → cosign [NEW]
│   └── Aggregation → DefectDojo [NEW]
│
├── Running Application
│   ├── API Endpoints → Newman + WireMock
│   ├── Contract Testing → Pact [NEW]
│   ├── Web UI → Playwright
│   ├── Accessibility → axe-core [NEW]
│   ├── Visual Regression → BackstopJS [NEW]
│   ├── Security (DAST) → ZAP + Nuclei
│   ├── API Fuzzing → RESTler [NEW]
│   └── Performance → K6/Locust/Artillery
│
├── Resilience Testing
│   ├── Network Chaos → Toxiproxy [NEW]
│   └── Runtime Security → Falco [NEW]
│
├── Database
│   ├── Migrations → Flyway [NEW]
│   └── Data Validation → Great Expectations [NEW]
│
└── AI-Assisted Testing
    └── Penetration Testing → CAI
```
