Bright Data - Structured Data Feeds
Extract structured data from major websites with automatic parsing. No scraping logic needed - just provide a URL and get clean JSON data.
Setup
Environment Variables (Required)
export BRIGHTDATA_API_KEY="your-api-key"
Optional
export BRIGHTDATA_POLLING_TIMEOUT=600 # Max seconds to wait (default: 600)
Get your API key from Bright Data Dashboard.
Usage
bash scripts/datasets.sh <dataset_type> <url> [additional_params...]
Available Datasets
E-Commerce
| Dataset | Command | Description |
|---|
| Amazon Product | datasets.sh amazon_product <url> | Product details, pricing, ratings |
| Amazon Reviews | datasets.sh amazon_product_reviews <url> | Customer reviews for a product |
| Amazon Search | datasets.sh amazon_product_search <keyword> <domain_url> | Search results |
| Walmart Product | datasets.sh walmart_product <url> | Product details from Walmart |
| Walmart Seller | datasets.sh walmart_seller <url> | Seller information |
| eBay Product | datasets.sh ebay_product <url> | eBay listing details |
| Home Depot | datasets.sh homedepot_products <url> | Home Depot product data |
| Zara | datasets.sh zara_products <url> | Zara product details |
| Etsy | datasets.sh etsy_products <url> | Etsy listing data |
| Best Buy | datasets.sh bestbuy_products <url> | Best Buy product info |
Professional Networks
| Dataset | Command | Description |
|---|
| LinkedIn Person | datasets.sh linkedin_person_profile <url> | Profile data (experience, skills) |
| LinkedIn Company | datasets.sh linkedin_company_profile <url> | Company page data |
| LinkedIn Jobs | datasets.sh linkedin_job_listings <url> | Job posting details |
| LinkedIn Posts | datasets.sh linkedin_posts <url> | Post content and engagement |
| LinkedIn Search | datasets.sh linkedin_people_search <url> <first> <last> | Find people |
| Crunchbase | datasets.sh crunchbase_company <url> | Company funding, employees |
| ZoomInfo | datasets.sh zoominfo_company_profile <url> | Company profile data |
Instagram
| Dataset | Command | Description |
|---|
| Profiles | datasets.sh instagram_profiles <url> | Bio, followers, following |
| Posts | datasets.sh instagram_posts <url> | Post details, likes, captions |
| Reels | datasets.sh instagram_reels <url> | Reel data and metrics |
| Comments | datasets.sh instagram_comments <url> | Post comments |
Facebook
| Dataset | Command | Description |
|---|
| Posts | datasets.sh facebook_posts <url> | Post content and reactions |
| Marketplace | datasets.sh facebook_marketplace_listings <url> | Listing details |
| Reviews | datasets.sh facebook_company_reviews <url> [num] | Company reviews |
| Events | datasets.sh facebook_events <url> | Event details |
TikTok
| Dataset | Command | Description |
|---|
| Profiles | datasets.sh tiktok_profiles <url> | Creator profile data |
| Posts | datasets.sh tiktok_posts <url> | Video details and metrics |
| Shop | datasets.sh tiktok_shop <url> | TikTok Shop product data |
| Comments | datasets.sh tiktok_comments <url> | Video comments |
YouTube
| Dataset | Command | Description |
|---|
| Profiles | datasets.sh youtube_profiles <url> | Channel data |
| Videos | datasets.sh youtube_videos <url> | Video details and stats |
| Comments | datasets.sh youtube_comments <url> [num] | Video comments (default: 10) |
Other Social
| Dataset | Command | Description |
|---|
| X (Twitter) | datasets.sh x_posts <url> | Tweet data |
| Reddit | datasets.sh reddit_posts <url> | Post and comment data |
Google Services
| Dataset | Command | Description |
|---|
| Maps Reviews | datasets.sh google_maps_reviews <url> [days] | Business reviews (default: 3 days) |
| Shopping | datasets.sh google_shopping <url> | Product comparison data |
| Play Store | datasets.sh google_play_store <url> | App details and reviews |
Other
| Dataset | Command | Description |
|---|
| Apple App Store | datasets.sh apple_app_store <url> | iOS app data |
| Reuters News | datasets.sh reuter_news <url> | News article content |
| GitHub | datasets.sh github_repository_file <url> | Repository file data |
| Yahoo Finance | datasets.sh yahoo_finance_business <url> | Stock and company data |
| Zillow | datasets.sh zillow_properties_listing <url> | Property listing details |
| Booking.com | datasets.sh booking_hotel_listings <url> | Hotel listing data |
Examples
Get LinkedIn Profile
bash scripts/datasets.sh linkedin_person_profile "https://www.linkedin.com/in/satyanadella/"
Get Amazon Product
bash scripts/datasets.sh amazon_product "https://www.amazon.com/dp/B09V3KXJPB"
Get Instagram Profile
bash scripts/datasets.sh instagram_profiles "https://www.instagram.com/natgeo/"
Get YouTube Comments
bash scripts/datasets.sh youtube_comments "https://www.youtube.com/watch?v=dQw4w9WgXcQ" 20
Search Amazon
bash scripts/datasets.sh amazon_product_search "wireless headphones" "https://www.amazon.com"
Output Format
Returns structured JSON with website-specific fields. Example for LinkedIn profile:
{
"name": "Satya Nadella",
"headline": "Chairman and CEO at Microsoft",
"location": "Greater Seattle Area",
"connections": "500+",
"experience": [...],
"education": [...],
"skills": [...]
}
How It Works
- Trigger: Sends URL to Bright Data's Web Data API
- Poll: Waits for data collection to complete (checks every second)
- Return: Outputs structured JSON when ready
The polling mechanism handles rate limits and ensures data quality by waiting for full extraction.
Advanced: Direct Fetch
For custom dataset IDs or advanced use cases:
bash scripts/fetch.sh <dataset_id> '<json_input>'
Example:
bash scripts/fetch.sh gd_l1viktl72bvl7bjuj0 '{"url":"https://linkedin.com/in/someone"}'