manoon-headless/scripts/gsc-monitoring

Google Search Console Monitoring Setup Guide

Overview

This setup creates an automated monitoring system for Google Search Console that runs daily and generates reports.

Prerequisites

1. Google Cloud account
2. Access to Google Search Console for manoonoils.com
3. kubectl access to your Kubernetes cluster

Authentication Methods

Choose one of the following authentication methods:

Option A: OAuth 2.0 (Recommended - No Service Account Key)

This is the easiest method if you can't create service account keys.

Step 1: Enable Search Console API

1. Go to https://console.cloud.google.com
2. Create/select project: manoonoils-monitoring
3. Go to APIs & Services → Library
4. Search: "Google Search Console API"
5. Click: Enable

Step 2: Create OAuth Credentials

1. Go to APIs & Services → Credentials
2. Click: Create Credentials → OAuth client ID
3. Click: Configure Consent Screen
4. User Type: External
5. Fill in:
   • App name: ManoonOils GSC Monitor
   • User support email: your email
   • Developer contact: your email
6. Click: Save and Continue (3 times)
7. Click: Back to Dashboard
8. Back on Credentials page
9. Click: Create Credentials → OAuth client ID
10. Application type: Desktop app
11. Name: GSC Desktop Client
12. Click: Create
13. Click: DOWNLOAD JSON

Step 3: Run Local Authorization

On your local machine (laptop):

# Go to the monitoring directory
cd scripts/gsc-monitoring

# Install dependencies
pip3 install google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client

# Run the OAuth setup
python3 setup-oauth-local.py

This will:

• Open a browser for you to authorize the app
• Generate a gsc-oauth-credentials.json file
• The refresh token never expires!

Step 4: Deploy to Kubernetes

# Copy the credentials to server
scp gsc-oauth-credentials.json doorwaysftw:/tmp/

# Create the secret
ssh doorwaysftw "kubectl create secret generic gsc-oauth-credentials \
  --namespace=manoonoils \
  --from-file=oauth-credentials.json=/tmp/gsc-oauth-credentials.json"

# Deploy the monitoring
ssh doorwaysftw "kubectl apply -f -" < cronjob-oauth.yaml

# Verify
ssh doorwaysftw "kubectl get cronjob gsc-monitoring-oauth -n manoonoils"

---

Option B: Service Account (Requires Key Creation)

Note: This only works if you can create service account keys in Google Cloud.

Setup Steps

Step 1: Create Google Cloud Project

1. Go to https://console.cloud.google.com
2. Click "Create Project" (or select existing)
3. Name it: manoonoils-monitoring
4. Note the Project ID

Step 2: Enable Search Console API

1. In your project, go to "APIs & Services" → "Library"
2. Search for "Google Search Console API"
3. Click "Enable"

Step 3: Create Service Account

1. Go to "IAM & Admin" → "Service Accounts"
2. Click "Create Service Account"
3. Name: gsc-monitor
4. Description: Monitoring service for Google Search Console
5. Click "Create and Continue"
6. Role: Select "Search Console Viewer" (or "Owner" if not available)
7. Click "Done"

Step 4: Create and Download Key

1. Click on the service account you just created
2. Go to "Keys" tab
3. Click "Add Key" → "Create New Key"
4. Select "JSON" format
5. Click "Create" - this downloads the key file
6. SAVE THIS FILE SECURELY - you cannot download it again!

Step 5: Add Service Account to Search Console

1. Go to https://search.google.com/search-console
2. Select your property: manoonoils.com
3. Click "Settings" (gear icon) → "Users and Permissions"
4. Click "Add User"
5. Enter the service account email (from the JSON key file, looks like: gsc-monitor@manoonoils-monitoring.iam.gserviceaccount.com)
6. Permission level: "Full"
7. Click "Add"

Step 6: Store Credentials in Kubernetes

On your server (doorwaysftw), run:

# Copy the JSON key file to the server
scp /path/to/service-account-key.json doorwaysftw:/tmp/

# Create the secret in Kubernetes
ssh doorwaysftw "kubectl create secret generic gsc-service-account \
  --namespace=manoonoils \
  --from-file=service-account.json=/tmp/service-account-key.json"

# Verify the secret was created
ssh doorwaysftw "kubectl get secret gsc-service-account -n manoonoils"

Step 7: Build and Deploy

# Build the Docker image
cd scripts/gsc-monitoring
docker build -t gcr.io/manoonoils/gsc-monitoring:latest .

# Push to registry (or use local registry)
docker push gcr.io/manoonoils/gsc-monitoring:latest

# Deploy to Kubernetes
kubectl apply -f cronjob.yaml

# Verify it's running
kubectl get cronjob gsc-monitoring -n manoonoils

Step 8: Test Manually

# Run a manual test
kubectl create job --from=cronjob/gsc-monitoring gsc-test -n manoonoils

# Check the logs
kubectl logs job/gsc-test -n manoonoils

# Delete the test job when done
kubectl delete job gsc-test -n manoonoils

What It Monitors

Daily Reports Include:

1. Search Analytics (Last 7 Days)

   • Total clicks and impressions
   • Average CTR and position
   • Top 5 search queries

2. Crawl Errors

   • Number of errors by type
   • Platform-specific issues

3. Sitemap Status

   • Sitemap processing status
   • Warnings and errors

Viewing Reports

Reports are saved to /var/log/gsc-monitoring/ in the pod and can be accessed:

# Get pod name
POD=$(kubectl get pods -n manoonoils -l job-name=gsc-monitoring -o name | head -1)

# View latest report
kubectl exec $POD -n manoonoils -- cat /var/log/gsc-monitoring/$(kubectl exec $POD -n manoonoils -- ls -t /var/log/gsc-monitoring/ | head -1)

Or set up log aggregation with your preferred tool.

Schedule

The monitoring runs daily at 9:00 AM UTC. To change:

# Edit the cronjob
kubectl edit cronjob gsc-monitoring -n manoonoils

# Change the schedule field (cron format)
# Examples:
# "0 */6 * * *"  # Every 6 hours
# "0 0 * * 0"    # Weekly on Sunday

Troubleshooting

"Service account key file not found"

• Verify the secret was created: kubectl get secret gsc-service-account -n manoonoils
• Check the key is mounted: kubectl exec deploy/gsc-monitoring -n manoonoils -- ls -la /etc/gsc-monitoring/

"User does not have permission"

• Verify the service account email was added to GSC with "Full" permissions
• Wait 5-10 minutes for permissions to propagate

"Site not found"

• Verify the SITE_URL in monitor.py matches exactly (with trailing slash)
• Check: https://search.google.com/search-console

Security Notes

• The service account JSON key is stored as a Kubernetes Secret
• The key has read-only access to Search Console data
• Rotate the key every 90 days for security
• Never commit the key file to git

Updating the Monitor

To update the monitoring script:

1. Edit monitor.py
2. Rebuild the Docker image
3. Push to registry
4. Delete and recreate the CronJob:

   kubectl delete cronjob gsc-monitoring -n manoonoils
   kubectl apply -f cronjob.yaml
   

Support

For issues or feature requests, check:

• Google Search Console API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3
• Google Cloud IAM docs: https://cloud.google.com/iam/docs