Compare commits
4 Commits
feature/do
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
044aefae94 | ||
|
|
36915a3f75 | ||
|
|
771e9dc20b | ||
|
|
df915ca128 |
@@ -8,7 +8,7 @@ spec:
|
||||
- web
|
||||
- websecure
|
||||
routes:
|
||||
- match: Host(`dev.manoonoils.com`) || Host(`manoonoils.com`) || Host(`www.manoonoils.com`)
|
||||
- match: Host(`manoonoils.com`) || Host(`www.manoonoils.com`)
|
||||
kind: Rule
|
||||
services:
|
||||
- name: storefront
|
||||
|
||||
16
scripts/gsc-monitoring/Dockerfile
Normal file
16
scripts/gsc-monitoring/Dockerfile
Normal file
@@ -0,0 +1,16 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install dependencies
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy monitoring script
|
||||
COPY monitor.py .
|
||||
|
||||
# Create log directory
|
||||
RUN mkdir -p /var/log/gsc-monitoring
|
||||
|
||||
# Run monitoring
|
||||
CMD ["python", "monitor.py"]
|
||||
121
scripts/gsc-monitoring/QUICKSTART.md
Normal file
121
scripts/gsc-monitoring/QUICKSTART.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# Google Search Console Monitoring Setup
|
||||
|
||||
## ✅ What's Been Created
|
||||
|
||||
I've created a complete automated monitoring system in `scripts/gsc-monitoring/`:
|
||||
|
||||
### Files Created:
|
||||
1. **monitor.py** - Python script that fetches GSC data
|
||||
2. **requirements.txt** - Python dependencies
|
||||
3. **Dockerfile** - Container image definition
|
||||
4. **cronjob.yaml** - Kubernetes CronJob for daily runs
|
||||
5. **README.md** - Full setup documentation
|
||||
|
||||
### What It Monitors:
|
||||
- ✅ Search analytics (clicks, impressions, CTR, position)
|
||||
- ✅ Top 5 search queries daily
|
||||
- ✅ Crawl errors
|
||||
- ✅ Sitemap status
|
||||
- ✅ Runs daily at 9 AM UTC
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Next Steps (Do These Now)
|
||||
|
||||
### Step 1: Create Google Cloud Project
|
||||
1. Go to https://console.cloud.google.com
|
||||
2. Create new project named `manoonoils-monitoring`
|
||||
3. Enable "Google Search Console API" in APIs & Services → Library
|
||||
|
||||
### Step 2: Create Service Account
|
||||
1. Go to IAM & Admin → Service Accounts
|
||||
2. Create service account: `gsc-monitor`
|
||||
3. Grant role: "Search Console Viewer" (or "Owner")
|
||||
|
||||
### Step 3: Download Key
|
||||
1. Click on the service account → Keys tab
|
||||
2. Add Key → Create New Key → JSON
|
||||
3. **Download and save the JSON file**
|
||||
|
||||
### Step 4: Add to Search Console
|
||||
1. Go to https://search.google.com/search-console
|
||||
2. Select `manoonoils.com` property
|
||||
3. Settings → Users and Permissions → Add User
|
||||
4. Add the service account email from the JSON file
|
||||
5. Permission level: "Full"
|
||||
|
||||
### Step 5: Deploy to Kubernetes
|
||||
|
||||
Run on your server:
|
||||
|
||||
```bash
|
||||
# Copy the JSON key to your server
|
||||
scp /path/to/downloaded-key.json doorwaysftw:/tmp/gsc-key.json
|
||||
|
||||
# Create the Kubernetes secret
|
||||
ssh doorwaysftw "kubectl create secret generic gsc-service-account \
|
||||
--namespace=manoonoils \
|
||||
--from-file=service-account.json=/tmp/gsc-key.json"
|
||||
|
||||
# Deploy the monitoring CronJob
|
||||
ssh doorwaysftw "kubectl apply -f -" < scripts/gsc-monitoring/cronjob.yaml
|
||||
|
||||
# Verify it's scheduled
|
||||
ssh doorwaysftw "kubectl get cronjob gsc-monitoring -n manoonoils"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Viewing Reports
|
||||
|
||||
### Check Latest Report:
|
||||
```bash
|
||||
ssh doorwaysftw "kubectl create job --from=cronjob/gsc-monitoring gsc-manual-test -n manoonoils
|
||||
sleep 10
|
||||
kubectl logs job/gsc-manual-test -n manoonoils
|
||||
kubectl delete job gsc-manual-test -n manoonoils"
|
||||
```
|
||||
|
||||
### Reports include:
|
||||
- Total clicks & impressions (last 7 days)
|
||||
- Average CTR and position
|
||||
- Top 5 search queries
|
||||
- Crawl errors summary
|
||||
- Sitemap status
|
||||
|
||||
---
|
||||
|
||||
## 🔒 Security
|
||||
|
||||
- Service account has **read-only** access to GSC
|
||||
- Credentials stored as Kubernetes Secret
|
||||
- JSON key never committed to git
|
||||
- Rotate key every 90 days
|
||||
|
||||
---
|
||||
|
||||
## 📚 Full Documentation
|
||||
|
||||
See `scripts/gsc-monitoring/README.md` for:
|
||||
- Detailed setup instructions
|
||||
- Troubleshooting guide
|
||||
- Updating the monitor
|
||||
- Changing schedule
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ Timeline
|
||||
|
||||
**Setup time:** 10-15 minutes
|
||||
**First report:** After setup (manual run) or next day (automatic)
|
||||
**Data availability:** 48-72 hours after setup (Google processes data)
|
||||
|
||||
---
|
||||
|
||||
## ❓ Questions?
|
||||
|
||||
The README.md has full troubleshooting. Common issues:
|
||||
- "User does not have permission" → Wait 5-10 min after adding to GSC
|
||||
- "Site not found" → Verify URL in monitor.py matches exactly
|
||||
|
||||
**Ready to proceed?** Start with Step 1 above!
|
||||
261
scripts/gsc-monitoring/README.md
Normal file
261
scripts/gsc-monitoring/README.md
Normal file
@@ -0,0 +1,261 @@
|
||||
# Google Search Console Monitoring Setup Guide
|
||||
|
||||
## Overview
|
||||
This setup creates an automated monitoring system for Google Search Console that runs daily and generates reports.
|
||||
|
||||
## Prerequisites
|
||||
1. Google Cloud account
|
||||
2. Access to Google Search Console for manoonoils.com
|
||||
3. kubectl access to your Kubernetes cluster
|
||||
|
||||
## Authentication Methods
|
||||
|
||||
Choose one of the following authentication methods:
|
||||
|
||||
### Option A: OAuth 2.0 (Recommended - No Service Account Key)
|
||||
|
||||
This is the **easiest method** if you can't create service account keys.
|
||||
|
||||
#### Step 1: Enable Search Console API
|
||||
1. Go to https://console.cloud.google.com
|
||||
2. Create/select project: `manoonoils-monitoring`
|
||||
3. Go to **APIs & Services → Library**
|
||||
4. Search: "Google Search Console API"
|
||||
5. Click: **Enable**
|
||||
|
||||
#### Step 2: Create OAuth Credentials
|
||||
1. Go to **APIs & Services → Credentials**
|
||||
2. Click: **Create Credentials → OAuth client ID**
|
||||
3. Click: **Configure Consent Screen**
|
||||
4. User Type: **External**
|
||||
5. Fill in:
|
||||
- App name: `ManoonOils GSC Monitor`
|
||||
- User support email: your email
|
||||
- Developer contact: your email
|
||||
6. Click: **Save and Continue** (3 times)
|
||||
7. Click: **Back to Dashboard**
|
||||
8. Back on Credentials page
|
||||
9. Click: **Create Credentials → OAuth client ID**
|
||||
10. Application type: **Desktop app**
|
||||
11. Name: `GSC Desktop Client`
|
||||
12. Click: **Create**
|
||||
13. Click: **DOWNLOAD JSON**
|
||||
|
||||
#### Step 3: Run Local Authorization
|
||||
On your local machine (laptop):
|
||||
|
||||
```bash
|
||||
# Go to the monitoring directory
|
||||
cd scripts/gsc-monitoring
|
||||
|
||||
# Install dependencies
|
||||
pip3 install google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client
|
||||
|
||||
# Run the OAuth setup
|
||||
python3 setup-oauth-local.py
|
||||
```
|
||||
|
||||
This will:
|
||||
- Open a browser for you to authorize the app
|
||||
- Generate a `gsc-oauth-credentials.json` file
|
||||
- The refresh token never expires!
|
||||
|
||||
#### Step 4: Deploy to Kubernetes
|
||||
|
||||
```bash
|
||||
# Copy the credentials to server
|
||||
scp gsc-oauth-credentials.json doorwaysftw:/tmp/
|
||||
|
||||
# Create the secret
|
||||
ssh doorwaysftw "kubectl create secret generic gsc-oauth-credentials \
|
||||
--namespace=manoonoils \
|
||||
--from-file=oauth-credentials.json=/tmp/gsc-oauth-credentials.json"
|
||||
|
||||
# Deploy the monitoring
|
||||
ssh doorwaysftw "kubectl apply -f -" < cronjob-oauth.yaml
|
||||
|
||||
# Verify
|
||||
ssh doorwaysftw "kubectl get cronjob gsc-monitoring-oauth -n manoonoils"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Option B: Service Account (Requires Key Creation)
|
||||
|
||||
**Note:** This only works if you can create service account keys in Google Cloud.
|
||||
|
||||
## Setup Steps
|
||||
|
||||
### Step 1: Create Google Cloud Project
|
||||
|
||||
1. Go to https://console.cloud.google.com
|
||||
2. Click "Create Project" (or select existing)
|
||||
3. Name it: `manoonoils-monitoring`
|
||||
4. Note the Project ID
|
||||
|
||||
### Step 2: Enable Search Console API
|
||||
|
||||
1. In your project, go to "APIs & Services" → "Library"
|
||||
2. Search for "Google Search Console API"
|
||||
3. Click "Enable"
|
||||
|
||||
### Step 3: Create Service Account
|
||||
|
||||
1. Go to "IAM & Admin" → "Service Accounts"
|
||||
2. Click "Create Service Account"
|
||||
3. Name: `gsc-monitor`
|
||||
4. Description: `Monitoring service for Google Search Console`
|
||||
5. Click "Create and Continue"
|
||||
6. Role: Select "Search Console Viewer" (or "Owner" if not available)
|
||||
7. Click "Done"
|
||||
|
||||
### Step 4: Create and Download Key
|
||||
|
||||
1. Click on the service account you just created
|
||||
2. Go to "Keys" tab
|
||||
3. Click "Add Key" → "Create New Key"
|
||||
4. Select "JSON" format
|
||||
5. Click "Create" - this downloads the key file
|
||||
6. **SAVE THIS FILE SECURELY** - you cannot download it again!
|
||||
|
||||
### Step 5: Add Service Account to Search Console
|
||||
|
||||
1. Go to https://search.google.com/search-console
|
||||
2. Select your property: `manoonoils.com`
|
||||
3. Click "Settings" (gear icon) → "Users and Permissions"
|
||||
4. Click "Add User"
|
||||
5. Enter the service account email (from the JSON key file, looks like: `gsc-monitor@manoonoils-monitoring.iam.gserviceaccount.com`)
|
||||
6. Permission level: "Full"
|
||||
7. Click "Add"
|
||||
|
||||
### Step 6: Store Credentials in Kubernetes
|
||||
|
||||
On your server (doorwaysftw), run:
|
||||
|
||||
```bash
|
||||
# Copy the JSON key file to the server
|
||||
scp /path/to/service-account-key.json doorwaysftw:/tmp/
|
||||
|
||||
# Create the secret in Kubernetes
|
||||
ssh doorwaysftw "kubectl create secret generic gsc-service-account \
|
||||
--namespace=manoonoils \
|
||||
--from-file=service-account.json=/tmp/service-account-key.json"
|
||||
|
||||
# Verify the secret was created
|
||||
ssh doorwaysftw "kubectl get secret gsc-service-account -n manoonoils"
|
||||
```
|
||||
|
||||
### Step 7: Build and Deploy
|
||||
|
||||
```bash
|
||||
# Build the Docker image
|
||||
cd scripts/gsc-monitoring
|
||||
docker build -t gcr.io/manoonoils/gsc-monitoring:latest .
|
||||
|
||||
# Push to registry (or use local registry)
|
||||
docker push gcr.io/manoonoils/gsc-monitoring:latest
|
||||
|
||||
# Deploy to Kubernetes
|
||||
kubectl apply -f cronjob.yaml
|
||||
|
||||
# Verify it's running
|
||||
kubectl get cronjob gsc-monitoring -n manoonoils
|
||||
```
|
||||
|
||||
### Step 8: Test Manually
|
||||
|
||||
```bash
|
||||
# Run a manual test
|
||||
kubectl create job --from=cronjob/gsc-monitoring gsc-test -n manoonoils
|
||||
|
||||
# Check the logs
|
||||
kubectl logs job/gsc-test -n manoonoils
|
||||
|
||||
# Delete the test job when done
|
||||
kubectl delete job gsc-test -n manoonoils
|
||||
```
|
||||
|
||||
## What It Monitors
|
||||
|
||||
### Daily Reports Include:
|
||||
|
||||
1. **Search Analytics** (Last 7 Days)
|
||||
- Total clicks and impressions
|
||||
- Average CTR and position
|
||||
- Top 5 search queries
|
||||
|
||||
2. **Crawl Errors**
|
||||
- Number of errors by type
|
||||
- Platform-specific issues
|
||||
|
||||
3. **Sitemap Status**
|
||||
- Sitemap processing status
|
||||
- Warnings and errors
|
||||
|
||||
## Viewing Reports
|
||||
|
||||
Reports are saved to `/var/log/gsc-monitoring/` in the pod and can be accessed:
|
||||
|
||||
```bash
|
||||
# Get pod name
|
||||
POD=$(kubectl get pods -n manoonoils -l job-name=gsc-monitoring -o name | head -1)
|
||||
|
||||
# View latest report
|
||||
kubectl exec $POD -n manoonoils -- cat /var/log/gsc-monitoring/$(kubectl exec $POD -n manoonoils -- ls -t /var/log/gsc-monitoring/ | head -1)
|
||||
```
|
||||
|
||||
Or set up log aggregation with your preferred tool.
|
||||
|
||||
## Schedule
|
||||
|
||||
The monitoring runs daily at **9:00 AM UTC**. To change:
|
||||
|
||||
```bash
|
||||
# Edit the cronjob
|
||||
kubectl edit cronjob gsc-monitoring -n manoonoils
|
||||
|
||||
# Change the schedule field (cron format)
|
||||
# Examples:
|
||||
# "0 */6 * * *" # Every 6 hours
|
||||
# "0 0 * * 0" # Weekly on Sunday
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Service account key file not found"
|
||||
- Verify the secret was created: `kubectl get secret gsc-service-account -n manoonoils`
|
||||
- Check the key is mounted: `kubectl exec deploy/gsc-monitoring -n manoonoils -- ls -la /etc/gsc-monitoring/`
|
||||
|
||||
### "User does not have permission"
|
||||
- Verify the service account email was added to GSC with "Full" permissions
|
||||
- Wait 5-10 minutes for permissions to propagate
|
||||
|
||||
### "Site not found"
|
||||
- Verify the SITE_URL in `monitor.py` matches exactly (with trailing slash)
|
||||
- Check: https://search.google.com/search-console
|
||||
|
||||
## Security Notes
|
||||
|
||||
- The service account JSON key is stored as a Kubernetes Secret
|
||||
- The key has read-only access to Search Console data
|
||||
- Rotate the key every 90 days for security
|
||||
- Never commit the key file to git
|
||||
|
||||
## Updating the Monitor
|
||||
|
||||
To update the monitoring script:
|
||||
|
||||
1. Edit `monitor.py`
|
||||
2. Rebuild the Docker image
|
||||
3. Push to registry
|
||||
4. Delete and recreate the CronJob:
|
||||
```bash
|
||||
kubectl delete cronjob gsc-monitoring -n manoonoils
|
||||
kubectl apply -f cronjob.yaml
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For issues or feature requests, check:
|
||||
- Google Search Console API docs: https://developers.google.com/webmaster-tools/search-console-api-original/v3
|
||||
- Google Cloud IAM docs: https://cloud.google.com/iam/docs
|
||||
32
scripts/gsc-monitoring/cronjob-oauth.yaml
Normal file
32
scripts/gsc-monitoring/cronjob-oauth.yaml
Normal file
@@ -0,0 +1,32 @@
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: gsc-monitoring-oauth
|
||||
namespace: manoonoils
|
||||
spec:
|
||||
schedule: "0 9 * * *" # Run daily at 9 AM UTC
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: gsc-monitor
|
||||
image: gcr.io/manoonoils/gsc-monitoring:latest
|
||||
env:
|
||||
- name: GSC_OAUTH_FILE
|
||||
value: /etc/gsc-monitoring/oauth-credentials.json
|
||||
- name: PYTHONUNBUFFERED
|
||||
value: "1"
|
||||
volumeMounts:
|
||||
- name: gsc-oauth-credentials
|
||||
mountPath: /etc/gsc-monitoring
|
||||
readOnly: true
|
||||
- name: logs
|
||||
mountPath: /var/log/gsc-monitoring
|
||||
volumes:
|
||||
- name: gsc-oauth-credentials
|
||||
secret:
|
||||
secretName: gsc-oauth-credentials
|
||||
- name: logs
|
||||
emptyDir: {}
|
||||
restartPolicy: OnFailure
|
||||
45
scripts/gsc-monitoring/cronjob.yaml
Normal file
45
scripts/gsc-monitoring/cronjob.yaml
Normal file
@@ -0,0 +1,45 @@
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: gsc-monitoring
|
||||
namespace: manoonoils
|
||||
spec:
|
||||
schedule: "0 9 * * *" # Run daily at 9 AM
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: gsc-monitor
|
||||
image: gcr.io/manoonoils/gsc-monitoring:latest
|
||||
env:
|
||||
- name: GSC_KEY_FILE
|
||||
value: /etc/gsc-monitoring/service-account.json
|
||||
- name: PYTHONUNBUFFERED
|
||||
value: "1"
|
||||
volumeMounts:
|
||||
- name: gsc-credentials
|
||||
mountPath: /etc/gsc-monitoring
|
||||
readOnly: true
|
||||
- name: logs
|
||||
mountPath: /var/log/gsc-monitoring
|
||||
volumes:
|
||||
- name: gsc-credentials
|
||||
secret:
|
||||
secretName: gsc-service-account
|
||||
- name: logs
|
||||
emptyDir: {}
|
||||
restartPolicy: OnFailure
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: gsc-service-account
|
||||
namespace: manoonoils
|
||||
type: Opaque
|
||||
stringData:
|
||||
service-account.json: |
|
||||
# PLACEHOLDER - Replace with actual service account JSON
|
||||
# Run: kubectl create secret generic gsc-service-account \
|
||||
# --namespace=manoonoils \
|
||||
# --from-file=service-account.json=/path/to/your/service-account-key.json
|
||||
234
scripts/gsc-monitoring/monitor.py
Normal file
234
scripts/gsc-monitoring/monitor.py
Normal file
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Google Search Console Monitoring Script
|
||||
Monitors search performance, crawl errors, and indexing status
|
||||
|
||||
Supports both:
|
||||
1. Service Account (with JSON key file)
|
||||
2. OAuth 2.0 (user authentication)
|
||||
"""
|
||||
|
||||
import os
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
from google.oauth2 import service_account
|
||||
from google.oauth2.credentials import Credentials as OAuthCredentials
|
||||
from google.auth.transport.requests import Request
|
||||
from googleapiclient.discovery import build
|
||||
from googleapiclient.errors import HttpError
|
||||
|
||||
# Configuration
|
||||
SITE_URL = "https://manoonoils.com/"
|
||||
SCOPES = ["https://www.googleapis.com/auth/webmasters.readonly"]
|
||||
KEY_FILE = os.environ.get("GSC_KEY_FILE", "/etc/gsc-monitoring/service-account.json")
|
||||
OAUTH_FILE = os.environ.get(
|
||||
"GSC_OAUTH_FILE", "/etc/gsc-monitoring/oauth-credentials.json"
|
||||
)
|
||||
|
||||
|
||||
def get_service():
|
||||
"""Authenticate and return Search Console service"""
|
||||
|
||||
# Try OAuth first
|
||||
if os.path.exists(OAUTH_FILE):
|
||||
print("Using OAuth authentication...")
|
||||
with open(OAUTH_FILE, "r") as f:
|
||||
creds_info = json.load(f)
|
||||
|
||||
creds = OAuthCredentials(
|
||||
token=creds_info["token"],
|
||||
refresh_token=creds_info["refresh_token"],
|
||||
token_uri=creds_info["token_uri"],
|
||||
client_id=creds_info["client_id"],
|
||||
client_secret=creds_info["client_secret"],
|
||||
scopes=creds_info["scopes"],
|
||||
)
|
||||
|
||||
# Refresh if expired
|
||||
if creds.expired:
|
||||
creds.refresh(Request())
|
||||
# Save updated credentials
|
||||
creds_info["token"] = creds.token
|
||||
with open(OAUTH_FILE, "w") as f:
|
||||
json.dump(creds_info, f, indent=2)
|
||||
|
||||
return build("webmasters", "v3", credentials=creds)
|
||||
|
||||
# Fall back to service account
|
||||
elif os.path.exists(KEY_FILE):
|
||||
print("Using Service Account authentication...")
|
||||
credentials = service_account.Credentials.from_service_account_file(
|
||||
KEY_FILE, scopes=SCOPES
|
||||
)
|
||||
return build("webmasters", "v3", credentials=credentials)
|
||||
|
||||
else:
|
||||
raise FileNotFoundError(
|
||||
f"No credentials found. Please set up either:\n"
|
||||
f" 1. OAuth: {OAUTH_FILE}\n"
|
||||
f" 2. Service Account: {KEY_FILE}\n"
|
||||
f"\nSee README.md for setup instructions."
|
||||
)
|
||||
|
||||
|
||||
def get_search_analytics(service, days=7):
|
||||
"""Get search analytics data for the last N days"""
|
||||
end_date = datetime.now().strftime("%Y-%m-%d")
|
||||
start_date = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%d")
|
||||
|
||||
try:
|
||||
request = {
|
||||
"startDate": start_date,
|
||||
"endDate": end_date,
|
||||
"dimensions": ["query", "page"],
|
||||
"rowLimit": 100,
|
||||
}
|
||||
|
||||
response = (
|
||||
service.searchanalytics().query(siteUrl=SITE_URL, body=request).execute()
|
||||
)
|
||||
|
||||
return response.get("rows", [])
|
||||
except HttpError as e:
|
||||
print(f"Error fetching search analytics: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def get_crawl_errors(service):
|
||||
"""Get crawl errors summary"""
|
||||
try:
|
||||
response = service.urlcrawlerrorscounts().query(siteUrl=SITE_URL).execute()
|
||||
return response.get("countPerTypes", [])
|
||||
except HttpError as e:
|
||||
print(f"Error fetching crawl errors: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def get_sitemaps(service):
|
||||
"""Get sitemap status"""
|
||||
try:
|
||||
response = service.sitemaps().list(siteUrl=SITE_URL).execute()
|
||||
return response.get("sitemap", [])
|
||||
except HttpError as e:
|
||||
print(f"Error fetching sitemaps: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def format_report(analytics, crawl_errors, sitemaps):
|
||||
"""Format monitoring report"""
|
||||
report = []
|
||||
report.append("=" * 70)
|
||||
report.append("GOOGLE SEARCH CONSOLE MONITORING REPORT")
|
||||
report.append(f"Site: {SITE_URL}")
|
||||
report.append(f"Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
report.append("=" * 70)
|
||||
|
||||
# Search Analytics Summary
|
||||
report.append("\n📊 SEARCH ANALYTICS (Last 7 Days)")
|
||||
report.append("-" * 70)
|
||||
|
||||
if analytics:
|
||||
total_clicks = sum(row["clicks"] for row in analytics)
|
||||
total_impressions = sum(row["impressions"] for row in analytics)
|
||||
avg_ctr = sum(row["ctr"] for row in analytics) / len(analytics) * 100
|
||||
avg_position = sum(row["position"] for row in analytics) / len(analytics)
|
||||
|
||||
report.append(f"Total Clicks: {total_clicks:,}")
|
||||
report.append(f"Total Impressions: {total_impressions:,}")
|
||||
report.append(f"Average CTR: {avg_ctr:.2f}%")
|
||||
report.append(f"Average Position: {avg_position:.1f}")
|
||||
|
||||
# Top 5 queries
|
||||
report.append("\n🔍 Top 5 Queries:")
|
||||
sorted_queries = sorted(analytics, key=lambda x: x["clicks"], reverse=True)[:5]
|
||||
for i, row in enumerate(sorted_queries, 1):
|
||||
query = row["keys"][0]
|
||||
clicks = row["clicks"]
|
||||
impressions = row["impressions"]
|
||||
report.append(
|
||||
f' {i}. "{query}" - {clicks} clicks, {impressions} impressions'
|
||||
)
|
||||
else:
|
||||
report.append("No search analytics data available yet (may take 48-72 hours)")
|
||||
|
||||
# Crawl Errors
|
||||
report.append("\n🚨 CRAWL ERRORS")
|
||||
report.append("-" * 70)
|
||||
|
||||
if crawl_errors:
|
||||
total_errors = sum(error.get("count", 0) for error in crawl_errors)
|
||||
if total_errors > 0:
|
||||
report.append(f"⚠️ Total Errors: {total_errors}")
|
||||
for error in crawl_errors:
|
||||
error_type = error.get("platform", "Unknown")
|
||||
category = error.get("category", "Unknown")
|
||||
count = error.get("count", 0)
|
||||
if count > 0:
|
||||
report.append(f" - {error_type} / {category}: {count}")
|
||||
else:
|
||||
report.append("✅ No crawl errors detected!")
|
||||
else:
|
||||
report.append("✅ No crawl errors detected!")
|
||||
|
||||
# Sitemaps
|
||||
report.append("\n🗺️ SITEMAPS")
|
||||
report.append("-" * 70)
|
||||
|
||||
if sitemaps:
|
||||
for sitemap in sitemaps:
|
||||
path = sitemap.get("path", "Unknown")
|
||||
is_pending = sitemap.get("isPending", False)
|
||||
is_sitemap_index = sitemap.get("isSitemapIndex", False)
|
||||
|
||||
status = "⏳ Pending" if is_pending else "✅ Processed"
|
||||
report.append(f" {path}")
|
||||
report.append(f" Status: {status}")
|
||||
|
||||
if not is_sitemap_index and "warnings" in sitemap:
|
||||
report.append(f" Warnings: {sitemap['warnings']}")
|
||||
if not is_sitemap_index and "errors" in sitemap:
|
||||
report.append(f" Errors: {sitemap['errors']} ⚠️")
|
||||
else:
|
||||
report.append(
|
||||
"⚠️ No sitemaps found. Submit your sitemap to Google Search Console!"
|
||||
)
|
||||
|
||||
report.append("\n" + "=" * 70)
|
||||
|
||||
return "\n".join(report)
|
||||
|
||||
|
||||
def main():
|
||||
"""Main monitoring function"""
|
||||
print("🔍 Starting Google Search Console monitoring...")
|
||||
|
||||
try:
|
||||
service = get_service()
|
||||
|
||||
# Gather data
|
||||
analytics = get_search_analytics(service)
|
||||
crawl_errors = get_crawl_errors(service)
|
||||
sitemaps = get_sitemaps(service)
|
||||
|
||||
# Generate and print report
|
||||
report = format_report(analytics, crawl_errors, sitemaps)
|
||||
print(report)
|
||||
|
||||
# Save report to file
|
||||
report_file = f"/var/log/gsc-monitoring/report_{datetime.now().strftime('%Y%m%d_%H%M%S')}.txt"
|
||||
os.makedirs(os.path.dirname(report_file), exist_ok=True)
|
||||
with open(report_file, "w") as f:
|
||||
f.write(report)
|
||||
print(f"\n💾 Report saved to: {report_file}")
|
||||
|
||||
except FileNotFoundError as e:
|
||||
print(f"❌ {e}")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"❌ Error: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
4
scripts/gsc-monitoring/requirements.txt
Normal file
4
scripts/gsc-monitoring/requirements.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
google-auth>=2.22.0
|
||||
google-auth-oauthlib>=1.0.0
|
||||
google-auth-httplib2>=0.1.1
|
||||
google-api-python-client>=2.95.0
|
||||
164
scripts/gsc-monitoring/setup-oauth-local.py
Normal file
164
scripts/gsc-monitoring/setup-oauth-local.py
Normal file
@@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
OAuth Setup for Google Search Console Monitoring
|
||||
Run this locally (not on the server) to generate OAuth credentials
|
||||
"""
|
||||
|
||||
import os
|
||||
import json
|
||||
import webbrowser
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def setup_oauth():
|
||||
"""Interactive OAuth setup"""
|
||||
|
||||
print("=" * 70)
|
||||
print("GOOGLE SEARCH CONSOLE - OAUTH 2.0 SETUP")
|
||||
print("=" * 70)
|
||||
print()
|
||||
print("This method uses OAuth 2.0 (no service account key needed)")
|
||||
print("You'll authenticate once with your Google account.")
|
||||
print()
|
||||
|
||||
# Step 1: Enable API
|
||||
print("STEP 1: Enable Search Console API")
|
||||
print("-" * 70)
|
||||
print("1. Go to: https://console.cloud.google.com")
|
||||
print("2. Create/select project: manoonoils-monitoring")
|
||||
print("3. Go to: APIs & Services → Library")
|
||||
print("4. Search: 'Google Search Console API'")
|
||||
print("5. Click: Enable")
|
||||
print()
|
||||
input("Press Enter when you've enabled the API...")
|
||||
|
||||
# Step 2: Create OAuth credentials
|
||||
print()
|
||||
print("STEP 2: Create OAuth Credentials")
|
||||
print("-" * 70)
|
||||
print("1. Go to: APIs & Services → Credentials")
|
||||
print("2. Click: Create Credentials → OAuth client ID")
|
||||
print("3. Click: Configure Consent Screen")
|
||||
print("4. User Type: External")
|
||||
print("5. App name: ManoonOils GSC Monitor")
|
||||
print("6. User support email: your-email@manoonoils.com")
|
||||
print("7. Developer contact: your-email@manoonoils.com")
|
||||
print("8. Click: Save and Continue (3 times)")
|
||||
print("9. Click: Back to Dashboard")
|
||||
print()
|
||||
print("10. Back on Credentials page:")
|
||||
print("11. Click: Create Credentials → OAuth client ID")
|
||||
print("12. Application type: Desktop app")
|
||||
print("13. Name: GSC Desktop Client")
|
||||
print("14. Click: Create")
|
||||
print("15. Click: DOWNLOAD JSON")
|
||||
print()
|
||||
|
||||
# Get the file path
|
||||
json_path = input("Enter the path to the downloaded JSON file: ").strip()
|
||||
|
||||
if not os.path.exists(json_path):
|
||||
print(f"❌ File not found: {json_path}")
|
||||
return
|
||||
|
||||
# Load credentials
|
||||
with open(json_path, "r") as f:
|
||||
client_config = json.load(f)
|
||||
|
||||
# Step 3: Install dependencies and run auth
|
||||
print()
|
||||
print("STEP 3: Install Dependencies")
|
||||
print("-" * 70)
|
||||
print("Run these commands:")
|
||||
print()
|
||||
print(
|
||||
" pip3 install google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client"
|
||||
)
|
||||
print()
|
||||
input("Press Enter after installing...")
|
||||
|
||||
# Step 4: Authorization
|
||||
print()
|
||||
print("STEP 4: Authorize Application")
|
||||
print("-" * 70)
|
||||
print("Running authorization...")
|
||||
|
||||
# Import here so we can check if installed
|
||||
try:
|
||||
from google_auth_oauthlib.flow import InstalledAppFlow
|
||||
from google.auth.transport.requests import Request
|
||||
import pickle
|
||||
except ImportError:
|
||||
print("❌ Please install the required packages first (Step 3)")
|
||||
return
|
||||
|
||||
SCOPES = ["https://www.googleapis.com/auth/webmasters.readonly"]
|
||||
|
||||
# Create flow
|
||||
flow = InstalledAppFlow.from_client_secrets_file(
|
||||
json_path,
|
||||
SCOPES,
|
||||
redirect_uri="urn:ietf:wg:oauth:2.0:oob", # For console-based auth
|
||||
)
|
||||
|
||||
# Get authorization URL
|
||||
auth_url, _ = flow.authorization_url(prompt="consent")
|
||||
|
||||
print()
|
||||
print("📱 Open this URL in your browser:")
|
||||
print(auth_url)
|
||||
print()
|
||||
|
||||
# Try to open browser automatically
|
||||
try:
|
||||
webbrowser.open(auth_url)
|
||||
print("(Browser should open automatically)")
|
||||
except:
|
||||
pass
|
||||
|
||||
# Get the code
|
||||
print()
|
||||
code = input("Enter the authorization code from the browser: ").strip()
|
||||
|
||||
# Exchange code for credentials
|
||||
flow.fetch_token(code=code)
|
||||
creds = flow.credentials
|
||||
|
||||
# Save credentials
|
||||
creds_info = {
|
||||
"token": creds.token,
|
||||
"refresh_token": creds.refresh_token,
|
||||
"token_uri": creds.token_uri,
|
||||
"client_id": creds.client_id,
|
||||
"client_secret": creds.client_secret,
|
||||
"scopes": creds.scopes,
|
||||
}
|
||||
|
||||
output_file = "gsc-oauth-credentials.json"
|
||||
with open(output_file, "w") as f:
|
||||
json.dump(creds_info, f, indent=2)
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print("✅ SUCCESS! OAuth credentials saved to:", output_file)
|
||||
print("=" * 70)
|
||||
print()
|
||||
print("NEXT STEPS:")
|
||||
print("1. Copy this file to your server:")
|
||||
print(f" scp {output_file} doorwaysftw:/tmp/")
|
||||
print()
|
||||
print("2. Create Kubernetes secret:")
|
||||
print(" ssh doorwaysftw")
|
||||
print(" kubectl create secret generic gsc-oauth-credentials \\")
|
||||
print(" --namespace=manoonoils \\")
|
||||
print(" --from-file=oauth-credentials.json=/tmp/gsc-oauth-credentials.json")
|
||||
print()
|
||||
print("3. Deploy monitoring:")
|
||||
print(" kubectl apply -f scripts/gsc-monitoring/cronjob-oauth.yaml")
|
||||
print()
|
||||
print("Your refresh token is valid indefinitely (until revoked).")
|
||||
print("The monitoring will run automatically every day!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
setup_oauth()
|
||||
133
scripts/gsc-monitoring/setup-oauth.py
Normal file
133
scripts/gsc-monitoring/setup-oauth.py
Normal file
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Google Search Console OAuth Setup Script
|
||||
Generates OAuth credentials and stores refresh token
|
||||
"""
|
||||
|
||||
import os
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def create_oauth_credentials():
|
||||
"""Guide user through OAuth setup"""
|
||||
|
||||
print("=" * 70)
|
||||
print("GOOGLE SEARCH CONSOLE - OAUTH SETUP (No Service Account Key Needed)")
|
||||
print("=" * 70)
|
||||
print()
|
||||
print("This method uses OAuth 2.0 instead of service account keys.")
|
||||
print("You'll authenticate once with your Google account.")
|
||||
print()
|
||||
|
||||
# Step 1: Create credentials
|
||||
print("STEP 1: Create OAuth Credentials")
|
||||
print("-" * 70)
|
||||
print("1. Go to: https://console.cloud.google.com")
|
||||
print("2. Select/create project: manoonoils-monitoring")
|
||||
print("3. Go to: APIs & Services → Credentials")
|
||||
print("4. Click: Create Credentials → OAuth client ID")
|
||||
print("5. Application type: Desktop app")
|
||||
print("6. Name: GSC Monitor")
|
||||
print("7. Click Create")
|
||||
print("8. Download the JSON file (client_secret_*.json)")
|
||||
print()
|
||||
input("Press Enter when you have downloaded the credentials file...")
|
||||
|
||||
# Step 2: Get credentials file path
|
||||
print()
|
||||
print("STEP 2: Upload Credentials")
|
||||
print("-" * 70)
|
||||
print("Copy the downloaded file to this server:")
|
||||
print()
|
||||
print(" scp /path/to/client_secret_*.json doorwaysftw:/tmp/gsc-credentials.json")
|
||||
print()
|
||||
input("Press Enter after uploading...")
|
||||
|
||||
# Step 3: Run authorization
|
||||
print()
|
||||
print("STEP 3: Authorize Application")
|
||||
print("-" * 70)
|
||||
print("Running authorization flow...")
|
||||
print()
|
||||
|
||||
# Create auth script
|
||||
auth_script = """#!/usr/bin/env python3
|
||||
import os
|
||||
import json
|
||||
import pickle
|
||||
from google_auth_oauthlib.flow import InstalledAppFlow
|
||||
from google.auth.transport.requests import Request
|
||||
|
||||
SCOPES = ['https://www.googleapis.com/auth/webmasters.readonly']
|
||||
CREDS_FILE = '/tmp/gsc-credentials.json'
|
||||
TOKEN_FILE = '/tmp/gsc-token.pickle'
|
||||
|
||||
def main():
|
||||
creds = None
|
||||
|
||||
if os.path.exists(TOKEN_FILE):
|
||||
with open(TOKEN_FILE, 'rb') as token:
|
||||
creds = pickle.load(token)
|
||||
|
||||
if not creds or not creds.valid:
|
||||
if creds and creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
else:
|
||||
flow = InstalledAppFlow.from_client_secrets_file(
|
||||
CREDS_FILE, SCOPES)
|
||||
creds = flow.run_local_server(port=0)
|
||||
|
||||
with open(TOKEN_FILE, 'wb') as token:
|
||||
pickle.dump(creds, token)
|
||||
|
||||
print("\\n✅ Authorization successful!")
|
||||
print(f"Token saved to: {TOKEN_FILE}")
|
||||
|
||||
# Save credentials info
|
||||
creds_info = {
|
||||
'token': creds.token,
|
||||
'refresh_token': creds.refresh_token,
|
||||
'token_uri': creds.token_uri,
|
||||
'client_id': creds.client_id,
|
||||
'client_secret': creds.client_secret,
|
||||
'scopes': creds.scopes
|
||||
}
|
||||
|
||||
with open('/tmp/gsc-token.json', 'w') as f:
|
||||
json.dump(creds_info, f, indent=2)
|
||||
|
||||
print(f"Credentials saved to: /tmp/gsc-token.json")
|
||||
print("\\nYou can now deploy the monitoring system!")
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
"""
|
||||
|
||||
# Save and run auth script
|
||||
with open("/tmp/gsc-auth.py", "w") as f:
|
||||
f.write(auth_script)
|
||||
|
||||
print("Authorization script created at: /tmp/gsc-auth.py")
|
||||
print()
|
||||
print("Run this on the server to authorize:")
|
||||
print()
|
||||
print(" ssh doorwaysftw")
|
||||
print(" cd /tmp")
|
||||
print(" python3 gsc-auth.py")
|
||||
print()
|
||||
print("This will open a browser for you to authorize the app.")
|
||||
print("If running on a remote server without browser, use SSH tunnel:")
|
||||
print()
|
||||
print(" ssh -L 8080:localhost:8080 doorwaysftw")
|
||||
print(" Then run python3 gsc-auth.py")
|
||||
print()
|
||||
|
||||
|
||||
def main():
|
||||
create_oauth_credentials()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,5 +1,5 @@
|
||||
import { createRouteHandler } from "@openpanel/nextjs/server";
|
||||
|
||||
export const { GET, POST } = createRouteHandler({
|
||||
apiUrl: "https://op.nodecrew.me/api",
|
||||
apiUrl: process.env.OPENPANEL_API_URL || "https://op.nodecrew.me/api",
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user