Automated Facebook Intelligence Scraper

Problem & Motivation

While managing product initiatives in the eMSP (e-Mobility Service Provider) sector at DenDen, our community operations team had to manually review competing charging network operators' Facebook pages every day to track tariff adjustments, station updates, and seasonal campaigns. This intelligence directly fueled customer service inquiries and in-app charging station guides. Manual monitoring was tedious and error-prone, occasionally causing app information to lag behind market reality.

The Solution

I architected and deployed an end-to-end automated market intelligence pipeline: scheduled agents scrape target social feeds, send raw posts to Gemini AI for schema-constrained extraction (e.g., post category, electricity tariff ranges, effective dates), and incrementally update a shared Google Sheet. Teams now maintain real-time market clarity without needing to manually browse Facebook.

Technical Stack & Anti-Detection

Given the lack of official Facebook feed APIs and strict bot detection mechanisms, traditional scraping libraries were routinely throttled. I utilized the Chrome DevTools Protocol (CDP) to programmatically direct a genuine Chrome instance, simulating organic human scroll velocities and retaining authenticated cookies to avoid challenge triggers. Raw post copy is processed via the Gemini API to turn messy unstructured text into validated tabular data, which is incrementally upserted into Google Sheets.

AI-Assisted Development Workflow

This project was built leveraging AI pair-programming and browser agent tooling. CDP lower-level controls, behavioral randomization, and Google Sheets differential write logic were accelerated with AI-assisted prototyping.

This collaboration enabled me to focus on high-leverage product decisions: defining operational pain points, choosing fields that produce actionable business value, and engineering resilient extraction prompts. While AI minimized implementation friction, problem definition, systems architecture, and output validation remained strictly human-driven.

Business Impact

The pipeline runs unattended end-to-end, saving 1 to 1.5 hours of repetitive manual tracking daily. Support personnel and app station listings now stay perpetually in sync with operator announcements. The historical dataset has also become an invaluable strategic asset for analyzing competitor tariff strategies and promotional frequency.

Modular Extensibility

The "Headless CDP + LLM Structuring + Cloud Sync" architecture is sector-agnostic. It can be effortlessly repurposed for retail discount tracking, dining reputation analysis, startup competitor research, or media brand monitoring.