feat: add Google Search Console automated monitoring

- Python monitoring script for daily GSC reports - Kubernetes CronJob for automated execution - Tracks search analytics, crawl errors, and sitemap status - Includes full setup documentation
2026-03-30 17:17:42 +02:00
parent 83efc4f1e2
commit df915ca128
5 changed files with 441 additions and 0 deletions
--- a/scripts/gsc-monitoring/README.md
+++ b/scripts/gsc-monitoring/README.md
@@ -0,0 +1,185 @@
+# Google Search Console Monitoring Setup Guide
+
+## Overview
+This setup creates an automated monitoring system for Google Search Console that runs daily and generates reports.
+
+## Prerequisites
+1. Google Cloud account
+2. Access to Google Search Console for manoonoils.com
+3. kubectl access to your Kubernetes cluster
+
+## Setup Steps
+
+### Step 1: Create Google Cloud Project
+
+1. Go to https://console.cloud.google.com
+2. Click "Create Project" (or select existing)
+3. Name it: `manoonoils-monitoring`
+4. Note the Project ID
+
+### Step 2: Enable Search Console API
+
+1. In your project, go to "APIs & Services" → "Library"
+2. Search for "Google Search Console API"
+3. Click "Enable"
+
+### Step 3: Create Service Account
+
+1. Go to "IAM & Admin" → "Service Accounts"
+2. Click "Create Service Account"
+3. Name: `gsc-monitor`
+4. Description: `Monitoring service for Google Search Console`
+5. Click "Create and Continue"
+6. Role: Select "Search Console Viewer" (or "Owner" if not available)
+7. Click "Done"
+
+### Step 4: Create and Download Key
+
+1. Click on the service account you just created
+2. Go to "Keys" tab
+3. Click "Add Key" → "Create New Key"
+4. Select "JSON" format
+5. Click "Create" - this downloads the key file
+6. **SAVE THIS FILE SECURELY** - you cannot download it again!
+
+### Step 5: Add Service Account to Search Console
+
+1. Go to https://search.google.com/search-console
+2. Select your property: `manoonoils.com`
+3. Click "Settings" (gear icon) → "Users and Permissions"
+4. Click "Add User"
+5. Enter the service account email (from the JSON key file, looks like: `gsc-monitor@manoonoils-monitoring.iam.gserviceaccount.com`)
+6. Permission level: "Full"
+7. Click "Add"
+
+### Step 6: Store Credentials in Kubernetes
+
+On your server (doorwaysftw), run:
+
+```bash
+# Copy the JSON key file to the server
+scp /path/to/service-account-key.json doorwaysftw:/tmp/
+
+# Create the secret in Kubernetes
+ssh doorwaysftw "kubectl create secret generic gsc-service-account \
+  --namespace=manoonoils \
+  --from-file=service-account.json=/tmp/service-account-key.json"
+
+# Verify the secret was created
+ssh doorwaysftw "kubectl get secret gsc-service-account -n manoonoils"
+```
+
+### Step 7: Build and Deploy
+
+```bash
+# Build the Docker image
+cd scripts/gsc-monitoring
+docker build -t gcr.io/manoonoils/gsc-monitoring:latest .
+
+# Push to registry (or use local registry)
+docker push gcr.io/manoonoils/gsc-monitoring:latest
+
+# Deploy to Kubernetes
+kubectl apply -f cronjob.yaml
+
+# Verify it's running
+kubectl get cronjob gsc-monitoring -n manoonoils
+```
+
+### Step 8: Test Manually
+
+```bash
+# Run a manual test
+kubectl create job --from=cronjob/gsc-monitoring gsc-test -n manoonoils
+
+# Check the logs
+kubectl logs job/gsc-test -n manoonoils
+
+# Delete the test job when done
+kubectl delete job gsc-test -n manoonoils
+```
+
+## What It Monitors
+
+### Daily Reports Include:
+
+1. **Search Analytics** (Last 7 Days)
+   - Total clicks and impressions
+   - Average CTR and position
+   - Top 5 search queries
+
+2. **Crawl Errors**
+   - Number of errors by type
+   - Platform-specific issues
+
+3. **Sitemap Status**
+   - Sitemap processing status
+   - Warnings and errors
+
+## Viewing Reports
+
+Reports are saved to `/var/log/gsc-monitoring/` in the pod and can be accessed:
+
+```bash
+# Get pod name
+POD=$(kubectl get pods -n manoonoils -l job-name=gsc-monitoring -o name | head -1)
+
+# View latest report
+kubectl exec $POD -n manoonoils -- cat /var/log/gsc-monitoring/$(kubectl exec $POD -n manoonoils -- ls -t /var/log/gsc-monitoring/ | head -1)
+```
+
+Or set up log aggregation with your preferred tool.
+
+## Schedule
+
+The monitoring runs daily at **9:00 AM UTC**. To change:
+
+```bash
+# Edit the cronjob
+kubectl edit cronjob gsc-monitoring -n manoonoils
+
+# Change the schedule field (cron format)
+# Examples:
+# "0 */6 * * *"  # Every 6 hours
+# "0 0 * * 0"    # Weekly on Sunday
+```
+
+## Troubleshooting
+
+### "Service account key file not found"
+- Verify the secret was created: `kubectl get secret gsc-service-account -n manoonoils`
+- Check the key is mounted: `kubectl exec deploy/gsc-monitoring -n manoonoils -- ls -la /etc/gsc-monitoring/`
+
+### "User does not have permission"
+- Verify the service account email was added to GSC with "Full" permissions
+- Wait 5-10 minutes for permissions to propagate
+
+### "Site not found"
+- Verify the SITE_URL in `monitor.py` matches exactly (with trailing slash)
+- Check: https://search.google.com/search-console
+
+## Security Notes
+
+- The service account JSON key is stored as a Kubernetes Secret
+- The key has read-only access to Search Console data
+- Rotate the key every 90 days for security
+- Never commit the key file to git
+
+## Updating the Monitor
+
+To update the monitoring script:
+
+1. Edit `monitor.py`
+2. Rebuild the Docker image
+3. Push to registry
+4. Delete and recreate the CronJob:
+   ```bash
+   kubectl delete cronjob gsc-monitoring -n manoonoils
+   kubectl apply -f cronjob.yaml
+   ```
+
+## Support
+
+For issues or feature requests, check:
+- Google Search Console API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3
+- Google Cloud IAM docs: https://cloud.google.com/iam/docs