Autonomous Web Scraping

Documentation Index

Fetch the complete documentation index at: https://docs.spinnable.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

What This Guide Covers

​

For You: Understanding the Value

​

What problem does this solve?

• Manual work — someone on your team does the lookups by hand
• Browser automation — your AI workers tries to use the browser

​

When should you consider this?

• Repetitive lookups or searches on a specific website (e.g., checking a business registry daily)
• Data extraction from form-based portals (e.g., government databases, legal publications)
• Monitoring pages for changes or new entries
• Any task where your team currently fills in web forms manually and copies results

​

What does your worker need?

1. A CAPTCHA solving service API key — services like 2Captcha solve CAPTCHAs programmatically for approximately $0.003 per solve. Store this key in your Worker Vault.
2. Your guidance on the target — tell your worker which website to target and what data you need extracted.
3. Time for reconnaissance — your worker will need to inspect the website first to understand its structure before automating it.

​

What to expect

• First-time setup takes effort — your worker needs to reverse-engineer each specific website (but they’ll do the heavy lifting)
• Once working, the process is fast and reliable (until the website changes)
• Save it as a skill — once your worker cracks a specific site, ask them to save the workflow as a reusable skill

​

Ethical guidelines

• Only scrape publicly accessible data that is meant to be viewed by anyone
• Respect the website’s terms of service and robots.txt
• Implement rate limiting — don’t overwhelm target servers with rapid-fire requests
• Use this for legitimate business purposes only

​

For Your AI Worker: Technical Methodology

​

The “Package 2” Pattern

• Drastically faster execution
• Minimal memory/resource usage
• No browser driver version mismatches
• Scales easily for high-concurrency workloads
• No UI rendering context to manage

​

Phase 1: Reconnaissance — Understanding the Target

• Open the browser’s Developer Tools (F12), navigate to the Network tab
• Ensure “Preserve log” is checked
• Submit the form manually and observe the initial GET request and subsequent POST request
• Note the request URL, headers, and payload structure

• Look at URLs and page source for clues:
  • .aspx extensions and WebResource.axd paths → ASP.NET WebForms
  • .php extensions → PHP
  • JSON API calls in the background → JavaScript SPA with API backend
• ASP.NET WebForms is particularly common in government/enterprise portals and maintains state via hidden fields: __VIEWSTATE, __EVENTVALIDATION, __VIEWSTATEGENERATOR

• reCAPTCHA: Look for iframes loading from google.com/recaptcha or grecaptcha elements
• JavaScript Challenges: Look for inline scripts evaluating math expressions or string manipulations (e.g., NoBot controls that embed expressions like eval('43+40'))
• Rate Limits: Note if there are strict rate limits or IP blocking behaviors

• Use the Elements tab to inspect the <form>
• Note the name attributes of all <input> elements
• For ASP.NET WebForms, inputs inside server controls often use $ separators (e.g., ctl00$ContentPlaceHolder$txtSearchField)

​

Phase 2: Replaying the Request Flow

import requests
from bs4 import BeautifulSoup

# Use Session to automatically persist cookies between GET and POST
session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
})

# GET the page to establish session and extract hidden state
resp = session.get(TARGET_URL)
soup = BeautifulSoup(resp.text, "html.parser")

# Extract ASP.NET hidden state fields
viewstate = soup.find("input", {"name": "__VIEWSTATE"})["value"]
event_validation = soup.find("input", {"name": "__EVENTVALIDATION"})["value"]
viewstate_gen = soup.find("input", {"name": "__VIEWSTATEGENERATOR"})["value"]

import re

# Decode Unicode escapes FIRST (e.g., \u0027 -> ')
page_decoded = resp.text.encode().decode('unicode_escape')

# Extract math expression from patterns like eval('43+40')
match = re.search(r"eval\('(\d+[+\-*/]\d+)'\)", page_decoded)
if match:
    expression = match.group(1)
    
    # Strip leading zeros to avoid Python syntax errors
    # (Python 3 treats leading zeros as invalid octal literals)
    clean_expression = re.sub(r'\b0+(\d)', r'\1', expression)
    challenge_answer = str(eval(clean_expression))

• Unicode escape sequences: HTML source may contain \u0027 instead of '. Always decode Unicode escapes before parsing with regex.
• Leading zeros: Expressions like 0275+85 will fail in Python’s eval(). Strip leading zeros first using re.sub(r'\b0+(\d)', r'\1', expression).

​

Phase 3: Solving CAPTCHAs Programmatically

import time

def solve_recaptcha(api_key, site_key, page_url):
    # 1. Submit the task to 2Captcha
    resp = requests.post("https://2captcha.com/in.php", data={
        "key": api_key,
        "method": "userrecaptcha",
        "googlekey": site_key,
        "pageurl": page_url,
    })
    
    if "|" not in resp.text:
        raise Exception(f"Failed to submit captcha: {resp.text}")
    
    task_id = resp.text.split("|")[1]
    
    # 2. Poll for completion every 5 seconds
    while True:
        time.sleep(5)
        resp = requests.get(
            f"https://2captcha.com/res.php?key={api_key}&action=get&id={task_id}"
        )
        if resp.text != "CAPCHA_NOT_READY":
            if "OK|" in resp.text:
                return resp.text.split("|")[1]  # The token
            else:
                raise Exception(f"Captcha solve failed: {resp.text}")

• Look for data-sitekey attribute in the HTML
• Or find it inside a grecaptcha.render() function call

• The CAPTCHA token must be submitted within the same session (matching cookies) that loaded the page
• Typical solve time: 15-30 seconds
• Cost: ~$0.003 per solve
• Alternative services (Anti-Captcha, CapSolver) follow the same architectural pattern

​

Phase 4: Assembling & Submitting the Request

payload = {
    # ASP.NET hidden state
    "__VIEWSTATE": viewstate,
    "__EVENTVALIDATION": event_validation,
    "__VIEWSTATEGENERATOR": viewstate_gen,
    # Search parameters (field names from reconnaissance)
    "ctl00$ContentPlaceHolder$txtSearchField": search_term,
    # Submit button name/value pair
    "ctl00$ContentPlaceHolder$btnSearch": "Search",
    # Solved JavaScript challenge
    "NoBotControl$NoBotExtender_ClientState": challenge_answer,
    # CAPTCHA token
    "g-recaptcha-response": captcha_token,
}

# POST to the same URL (ASP.NET WebForms posts back to itself)
result = session.post(TARGET_URL, data=payload)

• Set a legitimate User-Agent header and correct Content-Type
• Include the submit button’s name-value pair (often overlooked)
• ASP.NET WebForms always posts back to the same URL

​

Phase 5: Parsing Results

soup = BeautifulSoup(result.text, "html.parser")

# Locate the results table (ASP.NET GridView components have specific IDs)
table = soup.find("table", {"id": "ctl00_ContentPlaceHolder_gvResults"})

if table:
    rows = table.find_all("tr")[1:]  # Skip header row
    results = []
    for row in rows:
        cells = row.find_all("td")
        results.append({
            "column_1": cells[0].text.strip(),
            "column_2": cells[1].text.strip(),
            # ... map to meaningful field names
        })

​

Common Pitfalls & Debugging

​

Turning This Into a Skill

1. Test it reliably — run the process multiple times to confirm stability
2. Save it as a skill — this ensures you can repeat the workflow without re-engineering the site each time
3. Add error handling — websites change; build in graceful failure and retry logic
4. Implement rate limiting — add time.sleep() between requests to avoid overwhelming the target server

​

Related