What's Up? - Workflow Visualization

01

Project Overview

Understanding the autonomous blog engine

🤖

AI-Powered Content

Google Gemini 2.5 Flash generates rich, philosophical analysis posts with unique insights for every movie and series.

🎬

TMDB Integration

Automatically fetches movie/series data, cast info, and high-quality backdrop images from The Movie Database.

⚡

GitHub Actions

Scheduled workflows run twice daily, handling everything from content generation to deployment.

🖼️

Image Processing

All images are automatically processed to optimized WebP format with intelligent compression.

📧

Backup System

Automated backups every 3 days via email, with smart compression and split archives for large files.

🚀

Jekyll Deployment

Automatic build and deploy to GitHub Pages whenever new content is pushed to the repository.

Technology Stack

💎 Jekyll Static Site

🐙 GitHub Pages Hosting

⚙️ GitHub Actions Automation

🧠 Gemini 2.5 AI Content

🎞️ TMDB API Movie Data

📸 Unsplash Recap Images

✈️ Telegram Notifications

🐍 Python 3.11 Core Language

📅 Weekly Output Schedule

Monday - Saturday 4 posts/day 2 movies + 2 series (or 4 movies)

Sunday 1 post Weekly philosophical recap

Total 25 posts/week Fully automated

02

System Architecture

High-level flow of the automation system

⏰

GitHub Actions Triggers

0 3 * * * ~08:30 AM IST daily

0 12 * * 1-6 ~05:30 PM IST Mon-Sat

0 6 */3 * * ~11:30 AM IST (backup)

push to main Jekyll build & deploy

▼

🐍

main.py Execution

1 VALIDATE Environment

→

2 SELECT Items from CSV

→

3 FETCH TMDB Data

6 SAVE Post & Update

←

5 GENERATE Gemini Content

←

4 DOWNLOAD Process Images

7 REMOVE From CSV

→

8 PRE-CHECK Next Items

▼

📤

Git Commit & Push

Commits new posts, images, updated data files
Pushes to main branch
Triggers pages-deploy.yml workflow

▼

🚀

Jekyll Build & Deploy

Builds static site with Jekyll
Deploys to GitHub Pages
✨ Site is live!

03

Data Files Deep Dive

Core data structures powering the automation

📄

movies.csv

data/movies.csv

543 movies

Source list of movies to process (IMDb export format)

Key Columns Used:

`Const`	IMDb ID (unique identifier)	tt2488496
`Title`	Movie title	Star Wars: Episode VII
`Year`	Release year	2015
`Genres`	Genre list	Action, Adventure, Sci-Fi
`Directors`	Director name(s)	J.J. Abrams
`Runtime`	Duration	138 mins

Behavior:

Items are randomly selected (not sequential)
After processing, item is removed from CSV
Backup created at movies.backup.csv before each run

📺

series.csv

data/series.csv

~100 series

Source list of TV series to process (IMDb export format)

Depletion Handling:

                                if available_series.empty:

                                  print("⚠️ No more series - selecting 2nd movie!")

                                  # Select a second movie instead of series

When series.csv becomes empty, the system switches to double movie mode (processes 2 movies instead of 1 movie + 1 series)

📊

metadata_db.json

data/metadata_db.json

Last 100 entries

Tracks all generated posts for weekly recaps and analytics

{
  "posts": [
    {
      "imdb_id": "tt3748528",
      "title": "Rogue One: A Star Wars Story",
      "moods": ["Existential", "Heroic", "Melancholy"],
      "url": "/posts/rogue-one-a-star-wars-story/",
      "date": "2026-02-02T09:20:48.983494"
    }
  ]
}

Fields:

imdb_id - IMDb identifier
title - Movie/series title
moods - 3 mood tags extracted from content
url - Jekyll post URL
date - ISO timestamp of generation

📋

processing_queue.json

data/processing_queue.json

Pre-validated

Stores pre-validated items for next run (image availability confirmed)

1 At end of each run, system selects next items

2 Checks if TMDB has high-quality images (≥1920px)

3 If images available → saves to queue

4 Next run uses queued items (already validated)

Benefits:

⚡ Faster processing (no duplicate image checks)
🔔 Early warning for missing images
📧 Telegram/email alerts sent in advance

🖼️

used_images.json

data/used_images.json

Deduplication

Tracks all TMDB image paths ever used to ensure uniqueness

How Uniqueness Works:

When downloading images, load this file
Filter out already-used paths from TMDB backdrops
Select from remaining unused images
After saving, add new paths to this file

Additional Filters:

Landscape preference: aspect_ratio > 1.5
Text-free preference: iso_639_1 is None
Quality sorting: vote_average * width

📜

history.log

data/history.log

Permanent log

Permanent log of all processed items

Format:

# What's Up? Processing History
# Format: IMDB_ID  # Title - Date
tt3748528  # Rogue One - 2026-02-02 09:20
tt0944947  # Game of Thrones - 2026-02-02 09:21
recap_w6_2026  # Weekly Recap - Week 6

Uses:

Prevents re-processing of same items
Source for weekly recap (extracts posts from current week)
Audit trail

04

Core Python Scripts

The heart of the automation system

🐍

main.py

2,478 lines

Core Engine

⚙️

Configuration Section

Lines 1-80

API Keys (Environment Variables)

GEMINI_API_KEY
TMDB_API_KEY
UNSPLASH_ACCESS_KEY
TELEGRAM_BOT_TOKEN
TELEGRAM_CHAT_ID
SMTP_EMAIL / SMTP_PASSWORD

Image Configuration

HERO_MAX_SIZE_KB = 500
HERO_TARGET_WIDTH = 1920
BODY_MAX_SIZE_KB = 300
BODY_TARGET_WIDTH = 1280

✅

Validation Functions

Lines 81-130

validate_environment()

Checks required env vars: GEMINI_API_KEY, TMDB_API_KEY. Warns about optional vars. Exits with error if required vars missing.

validate_csv_files()

Verifies movies.csv and series.csv exist. Checks for required columns: Const, Title, Year. Prints item count.

📁

CSV & History Management

Lines 131-250

backup_csv_files()

Creates .backup.csv files before processing to prevent data loss.

remove_from_csv(imdb_id, media_type)

After processing, removes item from CSV to prevent re-selection.

load_history()

Reads history.log file. Returns set of processed IMDb IDs.

save_to_history(imdb_id, title)

Appends new entry with timestamp to history.log.

📅

Sunday Detection

Lines 251-290

def is_sunday():
    trigger = os.getenv('TRIGGER_SCHEDULE', '').strip()
    
    # Cron-based detection (authoritative on CI)
    if trigger == '0 14 * * 0':   # Sunday cron
        return True
    if trigger == '0 12 * * 1-6': # Mon-Sat cron
        return False
    
    # Fallback: day-of-week check
    return datetime.now().weekday() == 6

Uses TRIGGER_SCHEDULE env var from GitHub Actions. Falls back to system date for manual runs.

🎯

Item Selection

Lines 291-420

1

Load processing queue from previous run

Queue exists? → Use queued items (already validated)
No queue? → Select new items

2

Load both CSV files

Filter out already-processed items (from history). Count available items.

3

Random selection strategy

Pick 1 random movie + 1 random series (or 2nd movie if series depleted) + queue next items

🎬

TMDB API Integration

Lines 421-600

Step 1: Find TMDB ID from IMDb ID GET /find/{imdb_id}?external_source=imdb_id

Step 2: Detect media type (movie or TV) movie_results → 'movie' | tv_results → 'tv'

Step 3: Fetch full details GET /{media_type}/{tmdb_id}?append_to_response=images,credits,watch/providers

Data Retrieved:

Overview/plot summary
Cast list
Backdrop images (sorted by quality)
Streaming providers

🖼️

Image Processing

Lines 601-900

download_image(url, timeout=60)

Simple HTTP GET for image bytes. Returns raw image data or None.

process_and_save_image()

1. Open with PIL → 2. Convert to RGB → 3. Resize to target width → 4. Compress iteratively → 5. Save as WebP

download_and_process_images()

Handles deduplication, downloads hero + 3 body images with diversity algorithm.

🧠

Gemini AI Content Generation

Lines 1251-1800

Prompt Structure:

📋 METADATA - Title, Year, Type, Genres, Director, Runtime

🎬 TMDB DATA - Cast, Plot Overview

🔍 WEB SEARCH - Real critic reviews, balanced feedback

🖼️ IMAGE PLACEHOLDERS - [IMAGE_1], [IMAGE_2], [IMAGE_3]

✨ FORMATTING - Rich markdown, blockquotes with prompt styles

📝 STRUCTURE - Opening quote, sections, closing thought

Generation Settings:

temperature=0.85 - Creative but coherent
max_output_tokens=8192 - Long-form content
3 retry attempts with exponential backoff

📰

Weekly Recap Generation

Lines 1300-1650

Output Features:

1500-2000 word philosophical synthesis
Weaves ALL films from the week into one narrative
6-8 blockquotes from philosophers
Unsplash attribution in frontmatter

🚀

Main Entry Point

Lines 2350-2478

START

↓

Is it Sunday?

Yes

process_sunday_special()

Get week's posts
Fetch Unsplash
Generate recap
Save post

No

Normal Weekday Flow

validate_environment
validate_csv_files
backup_csv_files
select_items()
process_item(movie)
process_item(series)

↓

EXIT (Git commits changes)

💾

backup.py

327 lines

Backup System

Backup Directories:

_posts/Blog posts

assets/img/posts/Post images

data/CSVs, metadata

Original/CSV backups

Key Functions:

collect_stats() - Scans directories, counts files and bytes
create_archive() - Creates .tar.gz with max compression
split_archive() - Splits if > 24MB (Gmail limit)
send_backup_email() - Sends beautiful HTML email with attachment

05

GitHub Actions Workflows

Automated CI/CD pipeline configuration

📝

daily_post.yml

210 lines

Content Automation

Schedule:

0 3 * * * 08:30 AM IST daily (Sun: recap)

0 12 * * 1-6 05:30 PM IST Mon-Sat only

Steps:

📥 Checkout

🐍 Setup Python 3.11

📦 Install dependencies

⏳ Random delay (0-20 min)

📁 Create directories

📋 Verify CSVs

🤖 Run main.py

📝 Git config

📤 Commit changes

🚀 Push to main

💾

backup.yml

85 lines

Data Backup

Schedule:

0 6 */3 * * ~11:30 AM IST every 3 days

Steps:

📥 Checkout

🐍 Setup Python 3.11

📁 Verify directories

💾 Run backup.py

📊 Write summary

🚀

pages-deploy.yml

65 lines

Jekyll Deployment

Trigger:

push to main When _posts/ changes

Jobs:

Build Job

Setup Ruby 3.3
Configure Pages
Jekyll build
Upload artifact

→

Deploy Job

Download artifact
Deploy to Pages
Output URL

06

Daily Automation Flow

Complete sequence of operations for each run

⏰ Trigger Time: 08:30 AM IST or 05:30 PM IST

⚙️ GitHub Actions

🐍 main.py

🌐 External APIs

GitHub Actions

🚀 Trigger workflow

GitHub Actions

⏳ Random delay (0-20 min)

main.py

✅ Validate environment → APIs

main.py

📂 Load CSV files & processing queue

🔄 FOR EACH ITEM (2 items: movie + series)

1 Check Telegram uploads

2 Fetch TMDB data

3 Download & process images

4 Generate content (Gemini)

5 Save .md post

6 Update history & metadata

7 Remove from CSV

main.py

🔍 Pre-check next items & save queue → APIs

GitHub Actions

📤 Git add → commit → push

—

🚀 Triggers Jekyll build & deploy → pages-deploy.yml

07

Sunday Weekly Recap Flow

The philosophical synthesis process

1

🔍 Detect Sunday

is_sunday() returns True based on TRIGGER_SCHEDULE or datetime.weekday()

2

📋 Get Week's Posts

Parse history.log, filter Monday 00:00 → Sunday 23:59, exclude recap entries. Minimum 3 posts required.

3

🔎 Generate Search Query

Gemini analyzes titles → outputs 2-4 word philosophical query
Example: "existential contemplation"

4

📸 Fetch Unsplash Image

Search /search/photos?orientation=landscape, filter not in used_unsplash.json, select random from top 10

5

🖼️ Process Hero Image

Convert to RGB → Resize to 1920px → Compress to ≤500KB WebP → Save as week{N}_{year}_recap_hero.webp

6

🧠 Generate Recap Content

Gemini creates 1500-2000 word philosophical synthesis weaving ALL films into one narrative with 6-8 blockquotes

7

💾 Save Post

Filename: YYYY-MM-DD-weekly-recap-week-N.md
Sanitize title, write to _posts/

8

📜 Update History

Entry: recap_w{N}_{year}
Prevents duplicate recap generation

08

Backup System

Automated data protection every 3 days

💾

backup.py Execution

Every 3 days at 06:00 UTC (~11:30 AM IST)

1 Validate SMTP credentials

2 Scan directories: _posts/, assets/img/posts/, data/, Original/

3 Create archive: whatsup_backup_YYYY-MM-DD_HHMM.tar.gz (gzip level 9)

4

Check size:

< 24MB → Single file > 24MB → Split into .part01, .part02...

5 Send email(s) with HTML summary + archive attachment(s)

6 Cleanup temporary files

✉️

Subject: 💾 What's Up? Backup — Feb 15, 2026

📊 Backup Summary

Directory	Files	Size
_posts/	150	2.5 MB
assets/img/posts	600	180 MB
data/	6	1.2 MB
Original/	4	0.5 MB
Total	760	184.2 MB

📦 Archive Info

Size: 45.3 MB (gzip level 9)
Compression ratio: 75.4% smaller

🔄 Restore Instructions

Download .tar.gz attachment
tar -xzf whatsup_backup_*.tar.gz
Copy folders to project root

📎 whatsup_backup_2026-02-15_0600.tar.gz

09

API Integrations

External services powering the automation

🎬

TMDB API

Base URL: https://api.themoviedb.org/3

Endpoints:

/find/{imdb_id} Convert IMDb ID to TMDB ID

/movie/{id} Get movie details

/tv/{id} Get TV show details

Image CDN:

https://image.tmdb.org/t/p/original/{file_path}

🧠

Google Gemini API

SDK: google-genai Python package

Model: gemini-2.5-flash

Features Used:

Blog post generation (800-1200 words)
Weekly recap synthesis (1500-2000 words)
Unsplash search query generation
Image prompt generation

📸

Unsplash API

Base URL: https://api.unsplash.com

Endpoint:

/search/photos Search for images

Parameters:

query - philosophical query
orientation=landscape
content_filter=high

✈️

Telegram Bot API

Base URL: https://api.telegram.org/bot{token}

Endpoints:

/sendMessageSend alerts

/getUpdatesPoll for photos

/getFileDownload path

/deleteMessageCleanup

📧

Gmail SMTP

Server: smtp.gmail.com:587

Auth: App Password (not regular password)

Usage:

Backup email delivery
Alert notifications
Manual fallback requests

10

Error Handling & Fallbacks

Resilient systems for edge cases

🔄

API Error Handling

Gemini API:

3 retry attempts max
Exponential backoff on 503/429 errors
Wait: 10s → 20s → 40s between retries

TMDB API:

Graceful degradation on failure
Returns None for web search fallback

🖼️

Missing Images Fallback

Telegram Manual Upload System:

Pre-check identifies missing images
Sends Telegram + Email alerts
Upload with captions: HERO_{imdb_id}, IMG1_{imdb_id}...
Next run checks for uploads before TMDB

📺

Series Depletion

When series.csv is empty:

                            if available_series.empty:

                              # Switch to double movie mode

                              # Select 2 movies instead

🔁

Image Deduplication Reset

When all images for an item have been used:

                            if not available_backdrops:

                              print("All images used!")

                              available_backdrops = backdrops  # Reset

11

Environment Variables

Configuration secrets and settings

Required

Automation fails without these

GEMINI_API_KEY Google Gemini AI Google AI Studio

TMDB_API_KEY Movie database TMDB Account

Optional

Features disabled without these

UNSPLASH_ACCESS_KEY Sunday recap images Unsplash Developer

TELEGRAM_BOT_TOKEN Manual upload fallback BotFather

TELEGRAM_CHAT_ID Target chat for alerts Bot chat

SMTP_EMAIL Email sender address Gmail

SMTP_PASSWORD Email app password Gmail App Passwords

NOTIFICATION_EMAIL Recipient for alerts Any email

GitHub Actions

CI/CD specific variables

GH_PAT Personal Access Token for push

TRIGGER_SCHEDULE Cron expression that triggered run

12

File Structure Reference

Complete project organization

📁 WHATSUP/

📁 .github/workflows/

📄 daily_post.yml Content automation

📄 backup.yml Data backup

📄 pages-deploy.yml Jekyll deployment

📁 scripts/

🐍 main.py Core automation (2478 lines)

🐍 backup.py Backup system (327 lines)

📄 requirements.txt Python dependencies

📁 data/

📊 movies.csv Source movies

📊 series.csv Source series

📜 history.log Processed items log

📋 metadata_db.json Post metadata

📋 processing_queue.json Pre-validated items

📋 used_images.json TMDB deduplication

📋 used_unsplash.json Unsplash deduplication

📁 _posts/

📝 2026-02-02-rogue-one.md

📝 2026-02-02-game-of-thrones.md

... More markdown posts

📁 assets/img/posts/

🖼️ tt3748528_hero.webp

🖼️ tt3748528_1.webp

... More WebP images

⚙️ _config.yml Jekyll configuration

💎 Gemfile Ruby dependencies

🏠 index.html Site homepage

📖 README.md Project readme

13

Troubleshooting Guide

Common issues and solutions

❌

"No items available to process"

Cause: Both CSV files are empty or all items processed

Solution:

Add new items to movies.csv/series.csv
Check history.log for processed items

⚠️

"Gemini API error: 503/429"

Cause: API overloaded or rate limit

Solution:

Automatic retry with exponential backoff
Wait and rerun manually if persists

🖼️

"No high-quality images found"

Cause: TMDB has no landscape backdrops ≥1920px

Solution:

System sends Telegram/email alert
Upload manually with caption HERO_{imdb_id}

🔑

"Missing required environment variables"

Cause: GitHub Secrets not configured

Solution:

Go to Repository Settings → Secrets → Actions
Add missing secrets

📤

"Push rejected - no changes"

Cause: No new content generated

Solution:

Check workflow logs for errors
Verify CSV files have unprocessed items

📰

"Weekly recap: Only X posts this week"

Cause: Fewer than 3 posts generated this week

Solution:

Recap requires minimum 3 posts
Will skip if threshold not met

🛠️ Manual Intervention Commands

Run locally (testing):

cd scripts && python main.py

Skip random delay:

python main.py --skip-delay

Trigger workflow manually:

Actions tab → Select workflow → "Run workflow"