manoon-headless/scripts/gsc-monitoring/README.md

# Google Search Console Monitoring Setup Guide

## Overview
This setup creates an automated monitoring system for Google Search Console that runs daily and generates reports.

## Prerequisites
1. Google Cloud account
2. Access to Google Search Console for manoonoils.com
3. kubectl access to your Kubernetes cluster

## Authentication Methods

Choose one of the following authentication methods:

### Option A: OAuth 2.0 (Recommended - No Service Account Key)

This is the **easiest method** if you can't create service account keys.

#### Step 1: Enable Search Console API
1. Go to https://console.cloud.google.com
2. Create/select project: `manoonoils-monitoring`
3. Go to **APIs & Services → Library**
4. Search: "Google Search Console API"
5. Click: **Enable**

#### Step 2: Create OAuth Credentials
1. Go to **APIs & Services → Credentials**
2. Click: **Create Credentials → OAuth client ID**
3. Click: **Configure Consent Screen**
4. User Type: **External**
5. Fill in:
   - App name: `ManoonOils GSC Monitor`
   - User support email: your email
   - Developer contact: your email
6. Click: **Save and Continue** (3 times)
7. Click: **Back to Dashboard**
8. Back on Credentials page
9. Click: **Create Credentials → OAuth client ID**
10. Application type: **Desktop app**
11. Name: `GSC Desktop Client`
12. Click: **Create**
13. Click: **DOWNLOAD JSON**

#### Step 3: Run Local Authorization
On your local machine (laptop):

```bash
# Go to the monitoring directory
cd scripts/gsc-monitoring

# Install dependencies
pip3 install google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client

# Run the OAuth setup
python3 setup-oauth-local.py
```

This will:
- Open a browser for you to authorize the app
- Generate a `gsc-oauth-credentials.json` file
- The refresh token never expires!

#### Step 4: Deploy to Kubernetes

```bash
# Copy the credentials to server
scp gsc-oauth-credentials.json doorwaysftw:/tmp/

# Create the secret
ssh doorwaysftw "kubectl create secret generic gsc-oauth-credentials \
  --namespace=manoonoils \
  --from-file=oauth-credentials.json=/tmp/gsc-oauth-credentials.json"

# Deploy the monitoring
ssh doorwaysftw "kubectl apply -f -" < cronjob-oauth.yaml

# Verify
ssh doorwaysftw "kubectl get cronjob gsc-monitoring-oauth -n manoonoils"
```

---

### Option B: Service Account (Requires Key Creation)

**Note:** This only works if you can create service account keys in Google Cloud.

## Setup Steps

### Step 1: Create Google Cloud Project

1. Go to https://console.cloud.google.com
2. Click "Create Project" (or select existing)
3. Name it: `manoonoils-monitoring`
4. Note the Project ID

### Step 2: Enable Search Console API

1. In your project, go to "APIs & Services" → "Library"
2. Search for "Google Search Console API"
3. Click "Enable"

### Step 3: Create Service Account

1. Go to "IAM & Admin" → "Service Accounts"
2. Click "Create Service Account"
3. Name: `gsc-monitor`
4. Description: `Monitoring service for Google Search Console`
5. Click "Create and Continue"
6. Role: Select "Search Console Viewer" (or "Owner" if not available)
7. Click "Done"

### Step 4: Create and Download Key

1. Click on the service account you just created
2. Go to "Keys" tab
3. Click "Add Key" → "Create New Key"
4. Select "JSON" format
5. Click "Create" - this downloads the key file
6. **SAVE THIS FILE SECURELY** - you cannot download it again!

### Step 5: Add Service Account to Search Console

1. Go to https://search.google.com/search-console
2. Select your property: `manoonoils.com`
3. Click "Settings" (gear icon) → "Users and Permissions"
4. Click "Add User"
5. Enter the service account email (from the JSON key file, looks like: `gsc-monitor@manoonoils-monitoring.iam.gserviceaccount.com`)
6. Permission level: "Full"
7. Click "Add"

### Step 6: Store Credentials in Kubernetes

On your server (doorwaysftw), run:

```bash
# Copy the JSON key file to the server
scp /path/to/service-account-key.json doorwaysftw:/tmp/

# Create the secret in Kubernetes
ssh doorwaysftw "kubectl create secret generic gsc-service-account \
  --namespace=manoonoils \
  --from-file=service-account.json=/tmp/service-account-key.json"

# Verify the secret was created
ssh doorwaysftw "kubectl get secret gsc-service-account -n manoonoils"
```

### Step 7: Build and Deploy

```bash
# Build the Docker image
cd scripts/gsc-monitoring
docker build -t gcr.io/manoonoils/gsc-monitoring:latest .

# Push to registry (or use local registry)
docker push gcr.io/manoonoils/gsc-monitoring:latest

# Deploy to Kubernetes
kubectl apply -f cronjob.yaml

# Verify it's running
kubectl get cronjob gsc-monitoring -n manoonoils
```

### Step 8: Test Manually

```bash
# Run a manual test
kubectl create job --from=cronjob/gsc-monitoring gsc-test -n manoonoils

# Check the logs
kubectl logs job/gsc-test -n manoonoils

# Delete the test job when done
kubectl delete job gsc-test -n manoonoils
```

## What It Monitors

### Daily Reports Include:

1. **Search Analytics** (Last 7 Days)
   - Total clicks and impressions
   - Average CTR and position
   - Top 5 search queries

2. **Crawl Errors**
   - Number of errors by type
   - Platform-specific issues

3. **Sitemap Status**
   - Sitemap processing status
   - Warnings and errors

## Viewing Reports

Reports are saved to `/var/log/gsc-monitoring/` in the pod and can be accessed:

```bash
# Get pod name
POD=$(kubectl get pods -n manoonoils -l job-name=gsc-monitoring -o name | head -1)

# View latest report
kubectl exec $POD -n manoonoils -- cat /var/log/gsc-monitoring/$(kubectl exec $POD -n manoonoils -- ls -t /var/log/gsc-monitoring/ | head -1)
```

Or set up log aggregation with your preferred tool.

## Schedule

The monitoring runs daily at **9:00 AM UTC**. To change:

```bash
# Edit the cronjob
kubectl edit cronjob gsc-monitoring -n manoonoils

# Change the schedule field (cron format)
# Examples:
# "0 */6 * * *"  # Every 6 hours
# "0 0 * * 0"    # Weekly on Sunday
```

## Troubleshooting

### "Service account key file not found"
- Verify the secret was created: `kubectl get secret gsc-service-account -n manoonoils`
- Check the key is mounted: `kubectl exec deploy/gsc-monitoring -n manoonoils -- ls -la /etc/gsc-monitoring/`

### "User does not have permission"
- Verify the service account email was added to GSC with "Full" permissions
- Wait 5-10 minutes for permissions to propagate

### "Site not found"
- Verify the SITE_URL in `monitor.py` matches exactly (with trailing slash)
- Check: https://search.google.com/search-console

## Security Notes

- The service account JSON key is stored as a Kubernetes Secret
- The key has read-only access to Search Console data
- Rotate the key every 90 days for security
- Never commit the key file to git

## Updating the Monitor

To update the monitoring script:

1. Edit `monitor.py`
2. Rebuild the Docker image
3. Push to registry
4. Delete and recreate the CronJob:
   ```bash
   kubectl delete cronjob gsc-monitoring -n manoonoils
   kubectl apply -f cronjob.yaml
   ```

## Support

For issues or feature requests, check:
- Google Search Console API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3
- Google Cloud IAM docs: https://cloud.google.com/iam/docs