Apify Scrapers

Overview

Scrape content from major social platforms using Apify actors. Each platform has optimized settings for cost and quality.

Quick Decision Tree

What do you want to scrape? │ ├── Social Media Posts │ ├── Twitter/X → references/twitter.md │ │ └── Script: scripts/scrape_twitter_ai_trends.py │ │ │ ├── Reddit → references/reddit.md │ │ └── Script: scripts/scrape_reddit_ai_tech.py │ │ │ ├── LinkedIn → references/linkedin.md │ │ └── Script: scripts/scrape_linkedin_posts.py │ │ │ ├── Instagram → references/instagram.md │ │ └── Script: scripts/scrape_instagram.py │ │ └── Modes: profile, posts, hashtag, reels, comments │ │ │ ├── Facebook → references/facebook.md │ │ └── Script: scripts/scrape_facebook.py │ │ └── Modes: page, posts, reviews, groups, marketplace │ │ │ ├── TikTok → references/multi-platform.md │ │ └── Script: scripts/scrape_multi_platform.py │ │ │ └── YouTube → references/multi-platform.md │ └── Script: scripts/scrape_multi_platform.py │ ├── Business/Places │ ├── Google Maps businesses → references/google-maps.md │ │ └── Script: scripts/scrape_google_maps.py │ │ └── Modes: search, place, reviews │ │ │ └── Contact info from websites → references/contact-enrichment.md │ └── Script: scripts/scrape_contact_info.py │ └── Extract: emails, phone numbers, social profiles │ ├── Auto-detect URL type → references/url-detect.md │ └── Script: scripts/scrape_content_by_url.py │ ├── Trend Analysis (NEW) │ └── Enriched trend analysis → workflows/trend-analysis.md │ └── Script: scripts/analyze_trends.py │ └── Features: velocity scoring, lifecycle staging, opportunity scoring │ └── Workflows (multi-step) ├── Lead generation → workflows/lead-generation.md ├── Influencer discovery → workflows/influencer-discovery.md ├── Competitor analysis → workflows/competitor-intel.md ├── Trend analysis → workflows/trend-analysis.md └── Competitor Ads Intelligence (NEW) → workflows/competitor-ads.md └── Script: scripts/scrape_competitor_ads.py └── Platforms: Facebook Ads Library, Google Ads Transparency └── Features: Spend estimates, creative analysis, benchmarking

Environment Setup

Required in .env

APIFY_TOKEN=apify_api_xxxxx

Get your API key: https://console.apify.com/account/integrations

Common Usage Patterns

Scrape Twitter Trends

python scripts/scrape_twitter_ai_trends.py --query "AI agents" --max-tweets 50

Scrape Reddit Discussions

python scripts/scrape_reddit_ai_tech.py --subreddits "MachineLearning,LocalLLaMA" --max-posts 100

Scrape LinkedIn Author

python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/username" --max-posts 30

Auto-detect and Scrape URL

python scripts/scrape_content_by_url.py "https://x.com/user/status/123456"

Scrape Instagram Profile

python scripts/scrape_instagram.py profile "https://instagram.com/username" --max-posts 20

Scrape Instagram Hashtag

python scripts/scrape_instagram.py hashtag "#artificialintelligence" --max-posts 50

Scrape Instagram Reels

python scripts/scrape_instagram.py reels "https://instagram.com/username" --max-reels 30

Scrape Facebook Page

python scripts/scrape_facebook.py page "https://facebook.com/pagename" --max-posts 50

Scrape Facebook Reviews

python scripts/scrape_facebook.py reviews "https://facebook.com/pagename" --max-reviews 100

Scrape Facebook Marketplace

python scripts/scrape_facebook.py marketplace "laptops in san francisco" --max-items 30

Scrape Google Maps Businesses

python scripts/scrape_google_maps.py search "AI consulting firms in New York" --max-results 50

Scrape Google Maps Reviews

python scripts/scrape_google_maps.py reviews "ChIJN1t_tDeuEmsRUsoyG83frY4" --max-reviews 100

Extract Contact Info from Websites

python scripts/scrape_contact_info.py "https://example.com" --depth 2

Bulk Contact Enrichment

python scripts/scrape_contact_info.py --urls-file companies.txt --output contacts.json

Scrape Competitor Ads (Single Competitor)

python scripts/scrape_competitor_ads.py "Nike" --platforms facebook google --country US --days 30

Compare Multiple Competitors' Ads

python scripts/scrape_competitor_ads.py "Nike" "Adidas" "Puma" --compare --output comparison.json

Discover Advertisers by Keyword

python scripts/scrape_competitor_ads.py --search "running shoes" --country US --max-ads 200

Filter Competitor Ads by Media Type

python scripts/scrape_competitor_ads.py "Netflix" "Disney+" --platforms facebook --media-types video --days 7

Analyze Trends (NEW)

Analyze specific topic with enrichments

python scripts/analyze_trends.py "artificial intelligence" --sources google instagram tiktok --days 90

Discover trending topics in category

python scripts/analyze_trends.py --category technology --discover --top 50

Compare multiple trends

python scripts/analyze_trends.py "AI" "blockchain" "metaverse" --compare

Export HTML trend report

python scripts/analyze_trends.py "sustainable fashion" --format html --output trend_report.html

Cost Estimates

Platform Actor Cost per Item

Twitter kaitoeasyapi/twitter-x-data-tweet-scraper ~$0.00025

Reddit trudax/reddit-scraper ~$0.001-0.005

LinkedIn harvestapi/linkedin-post-search ~$0.01-0.05

YouTube streamers/youtube-scraper ~$0.01-0.05

TikTok clockworks/tiktok-scraper ~$0.005

Instagram (profile) apify/instagram-profile-scraper ~$0.005

Instagram (posts) apify/instagram-post-scraper ~$0.002-0.005

Instagram (hashtag) apify/instagram-hashtag-scraper ~$0.002-0.005

Instagram (reels) apify/instagram-reel-scraper ~$0.005-0.01

Instagram (comments) apify/instagram-comment-scraper ~$0.001-0.003

Facebook (page) apify/facebook-pages-scraper ~$0.005-0.01

Facebook (posts) apify/facebook-posts-scraper ~$0.003-0.005

Facebook (reviews) apify/facebook-reviews-scraper ~$0.002-0.005

Facebook (groups) apify/facebook-groups-scraper ~$0.005-0.01

Facebook (marketplace) apify/facebook-marketplace-scraper ~$0.005-0.01

Google Maps (search) compass/crawler-google-places ~$0.01-0.02

Google Maps (place) compass/google-maps-business-scraper ~$0.01

Google Maps (reviews) compass/google-maps-reviews-scraper ~$0.003-0.005

Contact Enrichment lukaskrivka/contact-info-scraper ~$0.01-0.03

Google Trends apify/google-trends-scraper ~$0.01

Trend Analysis (multi) Multiple actors ~$0.50-1.50/run

Facebook Ads Library apify/facebook-ads-scraper ~$0.75/1K ads

Facebook Ads (alt) curious_coder/facebook-ads-library-scraper ~$0.50/1K ads

Google Ads Transparency lexis-solutions/google-ads-scraper ~$1.00/1K ads

Google Ads (alt) xtech/google-ad-transparency-scraper ~$0.80/1K ads

Output Location

All scraped data saves to .tmp/ with timestamped filenames:

• .tmp/twitter_ai_trends_YYYYMMDD.json

• .tmp/reddit_ai_tech_YYYYMMDD.json

• .tmp/linkedin_posts_YYYYMMDD_HHMMSS.json

Security Notes

Credential Handling

• Store APIFY_TOKEN in .env file (never commit to git)

• Rotate API tokens periodically via Apify Console

• Never log or print API tokens in script output

• Use environment variables, not hardcoded values

Data Privacy

• Scraped data contains only publicly available content

• Social media posts may include PII (names, handles, profile info)

• Data is stored locally in .tmp/ directory

• No data is retained by Apify after actor run completes

• Consider data minimization - only scrape what you need

Access Scopes

• Apify tokens have full account access (no granular scopes)

• Use separate Apify accounts for different projects if needed

• Monitor usage via Apify Console dashboard

Compliance Considerations

• Terms of Service: Respect each platform's ToS (Twitter, Reddit, LinkedIn)

• Rate Limiting: Actors have built-in rate limiting to avoid bans

• Robots.txt: Some actors may bypass robots.txt - use responsibly

• GDPR: Scraped PII may be subject to GDPR if EU residents

• Ethical Use: Only scrape public data; never bypass authentication

• Proxy Ethics: Residential proxies should be used ethically

Troubleshooting

Common Issues

Issue: Actor run failed

Symptoms: Script terminates with "Actor run failed" or timeout error Cause: Invalid actor ID, insufficient proxy credits, or actor configuration issue Solution:

• Verify the actor ID is correct in the script

• Check Apify Console for actor run logs

• Ensure proxy settings match actor requirements

• Try running with default proxy settings first

Issue: Empty results returned

Symptoms: Script completes but returns 0 items Cause: Content blocked by platform, invalid query, or proxy being detected Solution:

• Try a different proxy type (residential vs datacenter)

• Simplify the search query

• Reduce the number of results requested

• Check if the platform is blocking scraping attempts

Issue: Rate limited by platform

Symptoms: Script fails with 429 errors or "rate limited" messages Cause: Too many requests in a short time period Solution:

• Add delays between requests (actor settings)

• Reduce concurrent requests

• Use proxy rotation

• Wait and retry after a cooldown period

Issue: Invalid API token

Symptoms: Authentication error or "invalid token" message Cause: Token expired, revoked, or incorrectly set Solution:

• Regenerate API token in Apify Console

• Verify token is correctly set in .env file

• Check for leading/trailing whitespace in token

• Ensure APIFY_TOKEN environment variable is loaded

Issue: Proxy connection errors

Symptoms: Connection timeout or proxy errors Cause: Proxy pool exhausted or geo-restriction issues Solution:

• Switch proxy type (basic, residential, or datacenter)

• Verify proxy credit balance in Apify Console

• Try a different proxy country/region

• Disable proxy to test if that's the root cause

Resources

Platform References

• references/twitter.md - Twitter/X scraping details

• references/reddit.md - Reddit scraping with subreddit targeting

• references/linkedin.md - LinkedIn post scraping (author or search mode)

• references/instagram.md - Instagram profile, posts, hashtag, reels, and comments scraping

• references/facebook.md - Facebook page, posts, reviews, groups, and marketplace scraping

• references/multi-platform.md - TikTok and YouTube scraping

• references/url-detect.md - Auto-detect URL type and scrape

Business/Places References

• references/google-maps.md - Google Maps business search, place details, and reviews

• references/contact-enrichment.md - Extract emails, phone numbers, and social profiles from websites

Workflow References

• workflows/lead-generation.md - Multi-step lead generation workflow

• workflows/influencer-discovery.md - Find and analyze influencers across platforms

• workflows/competitor-intel.md - Competitive intelligence gathering workflow

• workflows/trend-analysis.md - Enriched multi-platform trend analysis with scoring

Integration Patterns

Scrape and Enrich

Skills: apify-scrapers → parallel-research Use case: Scrape social media posts, then enrich with deep research Flow:

• Scrape Twitter/Reddit for mentions of a topic

• Extract company names or URLs from posts

• Use parallel-research to get detailed info on each company

Scrape and Summarize

Skills: apify-scrapers → content-generation Use case: Create newsletter content from social media trends Flow:

• Scrape trending AI posts from Twitter

• Pass scraped data to content-generation summarize

• Generate a formatted newsletter section

Scrape and Archive

Skills: apify-scrapers → google-workspace Use case: Save scraped data to Google Drive for team access Flow:

• Scrape LinkedIn posts from target accounts

• Format data as CSV or JSON

• Upload to Google Drive client folder via google-workspace

Trend Analysis + Content Strategy

Skills: apify-scrapers (trend-analysis) → content-generation Use case: Identify trending topics and create content strategy Flow:

• Run trend analysis: python scripts/analyze_trends.py "AI productivity" --sources all

• Review lifecycle stage and opportunity score

• Use content-generation to create content for high-opportunity trends

• Focus on emerging trends with high velocity scores

Competitive Trend Monitoring

Skills: apify-scrapers (trend-analysis) → parallel-research Use case: Monitor competitor visibility in trending topics Flow:

• Analyze industry trends: python scripts/analyze_trends.py --category "your-industry" --discover

• Compare your brand vs competitors in those trends

• Use parallel-research for deep dive on gaps

• Generate competitive intelligence report