Skip to main content

Error Handling Guide

Best practices for handling errors and implementing robust retry logic.

Overview

Web scraping is inherently prone to errors. Websites can be down, networks can fail, CAPTCHAs can appear unexpectedly, and dynamic content can take too long to load. Robust error handling is essential for building reliable scraping systems that can recover from failures and continue operating.

Browser7 provides detailed error information and status codes to help you diagnose and handle errors effectively. This guide covers common error types, best practices for error handling, and strategies for implementing retry logic.

SDK Error Classes

All Browser7 SDKs provide typed error classes (AuthenticationError, ValidationError, RateLimitError, InsufficientBalanceError, RenderError) that let you handle errors programmatically without parsing message strings. See the Node.js, Python, or PHP API reference for details.

Why Error Handling Matters

Production Reliability

In production systems, errors are inevitable:

  • Network Failures: Temporary connectivity issues
  • Website Changes: Page structure changes breaking selectors
  • Rate Limiting: Too many requests in a short time
  • Service Outages: Target websites or Browser7 temporarily unavailable

Without proper error handling, a single failure can crash your entire scraping job.

Cost Optimization

Not all failed renders are charged. User-caused failures (invalid selectors, target site errors, timeouts) are charged $0.01, but system errors (Browser7 infrastructure failures) are never charged. Check the billable field on failed renders to verify whether a charge was applied. See Error Codes Reference for the complete breakdown.

  • Detect Failures Early: Check status and errorCode before processing HTML
  • Implement Smart Retries: Don't retry unrecoverable errors; retry system errors immediately
  • Monitor Error Rates: Identify problematic sites and adjust strategies
  • Log Error Details: Understand why renders fail to improve success rates

Data Quality

Errors can result in incomplete or incorrect data:

  • Partial Renders: HTML captured before content loads
  • Missing Elements: Selectors don't match expected content
  • Stale Data: Cached responses instead of fresh data
  • CAPTCHA Blocks: Protected content not accessible

Proper error handling ensures you only process complete, accurate data.

Common Error Types

Network Errors

Connection failures before the render starts:

Symptoms:

  • 'Failed to connect to' in error message
  • Network timeouts
  • DNS resolution failures

Causes:

  • Poor network connectivity
  • Firewall or proxy issues
  • Browser7 API temporarily unreachable

Handling:

import { Browser7Error } from 'browser7';

try {
const result = await client.render(url);
} catch (error) {
if (error instanceof Browser7Error && !error.statusCode) {
console.error('Network error - retrying in 5 seconds...');
await delay(5000);
// Retry logic here
}
}

API Errors

Browser7 API rejects the request before rendering:

Symptoms:

  • 'Failed to start render' in error message
  • HTTP status codes (400, 401, 403, 429, 500)
  • Invalid parameters or authentication

Common causes:

  • 400: Invalid request parameters (bad URL, invalid country code, etc.)
  • 401: Invalid or missing API key
  • 403: Insufficient permissions or account suspended
  • 429: Rate limit exceeded
  • 500: Server error (temporary)

Handling:

import { AuthenticationError, RateLimitError, ValidationError } from 'browser7';

try {
const result = await client.render(url, options);
} catch (error) {
if (error instanceof AuthenticationError) {
console.error('Invalid API key');
// Don't retry - fix authentication
} else if (error instanceof RateLimitError) {
console.error('Rate limited - backing off...');
await delay(60000);
// Retry with backoff
} else if (error instanceof ValidationError) {
console.error('Invalid parameters:', error.details);
// Don't retry - fix parameters
}
}

Render Failures

Render starts but fails during execution:

Symptoms:

  • status: 'failed' in result
  • error field contains failure reason

Common causes:

  • Target website unreachable or down
  • Invalid URL (404, 500 errors from target site)
  • Timeout during page load
  • Browser crashes or resource exhaustion

Handling:

const result = await client.render(url);

if (result.status === 'failed') {
console.error('Render failed:', result.error);

// Check if retriable
if (result.error.includes('timeout')) {
// Retry with longer timeout or wait actions
} else if (result.error.includes('404')) {
// Don't retry - URL doesn't exist
}
}

Timeout Errors

Render exceeds maximum polling attempts:

Symptoms:

  • 'timed out after 60 attempts' in error message
  • Very long render times (>2 minutes)

Causes:

  • Extremely slow target website
  • Complex JavaScript taking too long to execute
  • Network issues delaying responses
  • Browser7 processing queue backup

Handling:

try {
const result = await client.render(url);
} catch (error) {
if (error.message.includes('timed out')) {
console.error('Render timed out - site may be too slow');
// Consider simplifying wait actions or using different approach
}
}

Wait Action Failures

Wait actions timeout or fail:

Symptoms:

  • result.timingBreakdown.waitActionsMs is very high (close to timeout)
  • Content missing from HTML
  • Status is 'completed' but data is incomplete

Causes:

  • Selectors don't match any elements
  • Elements never become visible
  • Text content never appears
  • Timeouts too short for slow-loading content

Handling:

const result = await client.render(url, { waitFor: actions });

if (result.timingBreakdown.waitActionsMs > 25000) {
console.warn('Wait actions took unusually long');
// Check if expected content exists in HTML
if (!result.html.includes('expected-content')) {
// Adjust wait actions or retry with different strategy
}
}

CAPTCHA Failures

CAPTCHA detected but not solved:

Symptoms:

  • result.captcha.detected: true
  • result.captcha.handled: false
  • HTML contains CAPTCHA widget

Causes:

  • CAPTCHA type not supported
  • CAPTCHA solving service temporarily unavailable
  • Complex CAPTCHA configuration
  • Site requires additional verification

Handling:

const result = await client.render(url, { captcha: 'auto' });

if (result.captcha.detected && !result.captcha.handled) {
console.error('CAPTCHA detected but not solved');
console.log('CAPTCHA type:', result.captcha.sitekey);
// Retry with specific CAPTCHA type or contact support
}

Best Practices

Always Use Try-Catch

Wrap all render calls in try-catch blocks:

async function safeFetch(url) {
try {
const result = await client.render(url);
return result;
} catch (error) {
console.error(`Error fetching ${url}:`, error.message);
return null;
}
}

Check Status Before Processing

Always verify status: 'completed' before using HTML:

const result = await client.render(url);

if (result.status !== 'completed') {
console.error('Render did not complete:', result.status);
if (result.error) {
console.error('Error:', result.error);
}
return null;
}

// Safe to process HTML
processHTML(result.html);

Implement Exponential Backoff

Retry with increasing delays between attempts:

async function renderWithRetry(url, maxRetries = 3) {
let lastError;

for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const result = await client.render(url);

if (result.status === 'completed') {
return result;
}

// Render failed, but no exception thrown
lastError = result.error;

} catch (error) {
lastError = error.message;
}

if (attempt < maxRetries) {
// Exponential backoff: 2s, 4s, 8s, etc.
const delayMs = Math.pow(2, attempt) * 1000;
console.log(`Retry ${attempt}/${maxRetries} in ${delayMs}ms...`);
await new Promise(resolve => setTimeout(resolve, delayMs));
}
}

throw new Error(`Failed after ${maxRetries} attempts: ${lastError}`);
}

Distinguish Retriable vs Non-Retriable Errors

Don't retry errors that won't succeed:

import { AuthenticationError, ValidationError, RateLimitError, Browser7Error } from 'browser7';

function isRetriable(error) {
// Non-retriable: authentication, validation, not found
if (error instanceof AuthenticationError) return false;
if (error instanceof ValidationError) return false;
if (error.statusCode === 404) return false;

// Retriable: rate limits (with delay), server errors, network errors
if (error instanceof RateLimitError) return true;
if (error.statusCode >= 500) return true;
if (error instanceof Browser7Error && !error.statusCode) return true;

// Unknown errors - retry cautiously
return true;
}

async function smartRetry(url, maxRetries = 3) {
let lastError;

for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
return await client.render(url);
} catch (error) {
lastError = error;

if (!isRetriable(error)) {
console.error('Non-retriable error:', error.message);
throw error; // Don't retry
}

if (attempt < maxRetries) {
await delay(Math.pow(2, attempt) * 1000);
}
}
}

throw lastError;
}

Log Comprehensive Error Details

Include context for debugging:

try {
const result = await client.render(url, options);

if (result.status === 'failed') {
console.error('Render failed:', {
url,
error: result.error,
selectedCity: result.selectedCity?.displayName,
timingBreakdown: result.timingBreakdown,
captcha: result.captcha
});
}

} catch (error) {
console.error('Exception during render:', {
url,
options,
error: error.message,
stack: error.stack
});
}

Set Reasonable Timeouts

Balance reliability with speed:

// Fast timeouts for quick fails (development)
Browser7.waitForSelector('.content', 'visible', 5000)

// Longer timeouts for reliability (production)
Browser7.waitForSelector('.content', 'visible', 30000)

// Very long for known slow sites
Browser7.waitForSelector('.content', 'visible', 60000)

Monitor Error Rates

Track success and failure rates:

const stats = {
total: 0,
success: 0,
failed: 0,
retried: 0,
errors: {}
};

async function monitoredRender(url) {
stats.total++;

try {
const result = await client.render(url);

if (result.status === 'completed') {
stats.success++;
return result;
} else {
stats.failed++;
stats.errors[result.error] = (stats.errors[result.error] || 0) + 1;
return null;
}
} catch (error) {
stats.failed++;
const errorKey = error.message.substring(0, 50);
stats.errors[errorKey] = (stats.errors[errorKey] || 0) + 1;
throw error;
}
}

// Periodic reporting
setInterval(() => {
console.log('Success rate:', (stats.success / stats.total * 100).toFixed(2) + '%');
console.log('Top errors:', Object.entries(stats.errors).sort((a, b) => b[1] - a[1]).slice(0, 5));
}, 60000);

Implement Circuit Breakers

Stop attempting when a site consistently fails:

class CircuitBreaker {
constructor(threshold = 5, timeout = 60000) {
this.failures = 0;
this.threshold = threshold;
this.timeout = timeout;
this.state = 'closed'; // closed, open, half-open
this.nextAttempt = null;
}

async execute(fn) {
if (this.state === 'open') {
if (Date.now() < this.nextAttempt) {
throw new Error('Circuit breaker is open');
}
this.state = 'half-open';
}

try {
const result = await fn();
this.onSuccess();
return result;
} catch (error) {
this.onFailure();
throw error;
}
}

onSuccess() {
this.failures = 0;
this.state = 'closed';
}

onFailure() {
this.failures++;
if (this.failures >= this.threshold) {
this.state = 'open';
this.nextAttempt = Date.now() + this.timeout;
console.error('Circuit breaker opened');
}
}
}

const breaker = new CircuitBreaker(5, 60000);

async function protectedRender(url) {
return breaker.execute(() => client.render(url));
}

Handling Specific Errors

Rate Limiting (429)

Respect rate limits with backoff:

async function handleRateLimit(url, retries = 3) {
for (let i = 0; i < retries; i++) {
try {
return await client.render(url);
} catch (error) {
if (error.message.includes('429')) {
const delayMs = Math.pow(2, i + 5) * 1000; // 32s, 64s, 128s
console.log(`Rate limited - waiting ${delayMs}ms...`);
await delay(delayMs);
} else {
throw error;
}
}
}
throw new Error('Rate limit persists after retries');
}

Authentication Errors (401)

Verify and refresh credentials:

async function renderWithAuth(url) {
try {
return await client.render(url);
} catch (error) {
if (error.message.includes('401')) {
console.error('Authentication failed - check API key');
// Fetch fresh API key from secure storage
// Create new client
throw new Error('Invalid API credentials');
}
throw error;
}
}

Target Site Errors (404, 500)

Handle target website issues:

const result = await client.render(url);

if (result.status === 'failed') {
if (result.error.includes('404')) {
console.error('URL not found:', url);
// Mark URL as invalid, don't retry
await markUrlInvalid(url);
return null;
} else if (result.error.includes('500')) {
console.error('Target site error - retrying...');
// Target site issue, retry after delay
await delay(10000);
return renderWithRetry(url);
}
}

Wait Action Timeouts

Adjust wait actions based on failures:

async function adaptiveRender(url, options) {
try {
const result = await client.render(url, options);

if (result.timingBreakdown.waitActionsMs > 25000) {
console.warn('Wait actions slow - adjusting timeouts');

// Increase timeouts for next attempt
options.waitFor = options.waitFor.map(action => ({
...action,
timeout: (action.timeout || 30000) * 1.5
}));
}

return result;
} catch (error) {
if (error.message.includes('timed out')) {
// Simplify wait actions
options.waitFor = options.waitFor.slice(0, 2);
return client.render(url, options);
}
throw error;
}
}

Retry Strategies

Simple Retry

Basic retry with fixed delay:

async function simpleRetry(url, attempts = 3, delayMs = 2000) {
for (let i = 0; i < attempts; i++) {
try {
return await client.render(url);
} catch (error) {
if (i === attempts - 1) throw error;
await delay(delayMs);
}
}
}

Exponential Backoff

Increasing delays between retries:

async function exponentialBackoff(url, maxAttempts = 5) {
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await client.render(url);
} catch (error) {
if (attempt === maxAttempts - 1) throw error;

const delayMs = Math.min(
Math.pow(2, attempt) * 1000,
32000 // Max 32 seconds
);

console.log(`Attempt ${attempt + 1} failed, retrying in ${delayMs}ms`);
await delay(delayMs);
}
}
}

Jittered Backoff

Add randomness to prevent thundering herd:

async function jitteredBackoff(url, maxAttempts = 5) {
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await client.render(url);
} catch (error) {
if (attempt === maxAttempts - 1) throw error;

const baseDelay = Math.pow(2, attempt) * 1000;
const jitter = Math.random() * 1000;
const delayMs = Math.min(baseDelay + jitter, 32000);

await delay(delayMs);
}
}
}

Conditional Retry

Only retry specific error types:

async function conditionalRetry(url, maxAttempts = 3) {
const retriableErrors = ['timeout', 'ECONNREFUSED', '500', '502', '503'];

for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
const result = await client.render(url);

if (result.status === 'completed') {
return result;
}

// Check if error is retriable
const isRetriable = retriableErrors.some(err =>
result.error?.includes(err)
);

if (!isRetriable || attempt === maxAttempts - 1) {
throw new Error(result.error);
}

await delay(Math.pow(2, attempt) * 1000);

} catch (error) {
const isRetriable = retriableErrors.some(err =>
error.message.includes(err)
);

if (!isRetriable || attempt === maxAttempts - 1) {
throw error;
}

await delay(Math.pow(2, attempt) * 1000);
}
}
}

Common Use Cases

Robust Scraper with Full Error Handling

async function robustScraper(urls) {
const results = [];
const errors = [];

for (const url of urls) {
try {
const result = await exponentialBackoff(url, 3);

if (result.status === 'completed') {
results.push({
url,
html: result.html,
city: result.selectedCity.displayName,
timing: result.timingBreakdown.totalMs
});
} else {
errors.push({ url, error: result.error });
}

} catch (error) {
errors.push({ url, error: error.message });
}
}

console.log(`Success: ${results.length}/${urls.length}`);
console.log(`Errors: ${errors.length}/${urls.length}`);

return { results, errors };
}

Production-Grade Error Handler

class ProductionRenderer {
constructor(client, options = {}) {
this.client = client;
this.maxRetries = options.maxRetries || 3;
this.breaker = new CircuitBreaker();
this.stats = { success: 0, failed: 0, retried: 0 };
}

async render(url, options = {}) {
return this.breaker.execute(async () => {
let lastError;

for (let attempt = 0; attempt < this.maxRetries; attempt++) {
try {
const result = await this.client.render(url, options);

if (result.status === 'completed') {
this.stats.success++;
if (attempt > 0) this.stats.retried++;
return result;
}

lastError = result.error;

if (!this.isRetriable(lastError)) {
throw new Error(lastError);
}

} catch (error) {
lastError = error.message;

if (!this.isRetriable(lastError)) {
this.stats.failed++;
throw error;
}
}

if (attempt < this.maxRetries - 1) {
await this.delay(Math.pow(2, attempt) * 1000);
}
}

this.stats.failed++;
throw new Error(`Failed after ${this.maxRetries} attempts: ${lastError}`);
});
}

isRetriable(error) {
const nonRetriable = ['400', '401', '403', '404'];
return !nonRetriable.some(code => error.includes(code));
}

async delay(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}

getStats() {
return this.stats;
}
}

SDK Implementations

See language-specific examples for error handling:

Python Examples

import time
from browser7 import Browser7

client = Browser7(api_key='b7_your_api_key_here')

# Basic try/except
try:
result = client.render('https://example.com')
print(result.html)
except Exception as e:
print(f'Error: {e}')

# Check status before processing
result = client.render('https://example.com')

if result.status != 'completed':
print(f'Render did not complete: {result.status}')
if result.error:
print(f'Error: {result.error}')
else:
process_html(result.html)

# Retry with exponential backoff
def render_with_retry(url, max_retries=3):
last_error = None

for attempt in range(1, max_retries + 1):
try:
result = client.render(url)
if result.status == 'completed':
return result
last_error = result.error
except Exception as e:
last_error = str(e)

if attempt < max_retries:
delay = (2 ** attempt) # 2s, 4s, 8s
print(f'Retry {attempt}/{max_retries} in {delay}s...')
time.sleep(delay)

raise Exception(f'Failed after {max_retries} attempts: {last_error}')

# Distinguish retriable vs non-retriable errors
NON_RETRIABLE = ['401', '400', '403', '404']

def is_retriable(error):
return not any(code in str(error) for code in NON_RETRIABLE)

def smart_retry(url, max_retries=3):
last_error = None

for attempt in range(max_retries):
try:
result = client.render(url)
if result.status == 'completed':
return result
last_error = result.error
if not is_retriable(last_error):
raise Exception(last_error)
except Exception as e:
if not is_retriable(str(e)):
raise
last_error = str(e)

if attempt < max_retries - 1:
time.sleep(2 ** (attempt + 1))

raise Exception(f'Failed after {max_retries} attempts: {last_error}')

PHP Examples

<?php

use Browser7\Browser7Client;

$client = new Browser7Client('b7_your_api_key_here');

// Basic try/catch
try {
$result = $client->render('https://example.com');
echo $result->html;
} catch (\RuntimeException $e) {
echo 'Error: ' . $e->getMessage();
}

// Check status before processing
$result = $client->render('https://example.com');

if ($result->status !== 'completed') {
echo 'Render did not complete: ' . $result->status;
if ($result->error) {
echo 'Error: ' . $result->error;
}
} else {
processHtml($result->html);
}

// Retry with exponential backoff
function renderWithRetry(Browser7Client $client, string $url, int $maxRetries = 3): mixed
{
$lastError = null;

for ($attempt = 1; $attempt <= $maxRetries; $attempt++) {
try {
$result = $client->render($url);
if ($result->status === 'completed') {
return $result;
}
$lastError = $result->error;
} catch (\RuntimeException $e) {
$lastError = $e->getMessage();
}

if ($attempt < $maxRetries) {
$delay = pow(2, $attempt); // 2s, 4s, 8s
echo "Retry {$attempt}/{$maxRetries} in {$delay}s...\n";
sleep($delay);
}
}

throw new \RuntimeException("Failed after {$maxRetries} attempts: {$lastError}");
}

// Distinguish retriable vs non-retriable errors
function isRetriable(string $error): bool
{
$nonRetriable = ['400', '401', '403', '404'];
foreach ($nonRetriable as $code) {
if (str_contains($error, $code)) {
return false;
}
}
return true;
}

function smartRetry(Browser7Client $client, string $url, int $maxRetries = 3): mixed
{
$lastError = null;

for ($attempt = 0; $attempt < $maxRetries; $attempt++) {
try {
$result = $client->render($url);
if ($result->status === 'completed') {
return $result;
}
$lastError = $result->error;
if (!isRetriable($lastError)) {
throw new \RuntimeException($lastError);
}
} catch (\RuntimeException $e) {
if (!isRetriable($e->getMessage())) {
throw $e;
}
$lastError = $e->getMessage();
}

if ($attempt < $maxRetries - 1) {
sleep(pow(2, $attempt + 1));
}
}

throw new \RuntimeException("Failed after {$maxRetries} attempts: {$lastError}");
}

Troubleshooting

High Error Rates

If you're experiencing frequent errors:

  • Check result.error messages to identify patterns
  • Monitor result.timingBreakdown for timeouts
  • Verify target websites are accessible independently
  • Check Browser7 status page for service issues
  • Review rate limiting and adjust request frequency
  • Simplify wait actions to reduce timeouts

Renders Always Fail for Specific Sites

If certain sites consistently fail:

  • Test site accessibility in a regular browser
  • Check if site has anti-bot protections (Cloudflare, etc.)
  • Try different geo-targeting locations
  • Enable CAPTCHA solving if not already enabled
  • Simplify or remove wait actions
  • Contact support with specific site details

Intermittent Failures

If errors occur randomly:

  • Implement retry logic with exponential backoff
  • Check network connectivity and stability
  • Monitor target site reliability (some sites are unreliable)
  • Add circuit breakers to prevent cascade failures
  • Increase wait action timeouts for reliability

Retries Not Helping

If retries don't improve success rates:

  • Verify you're retrying the right error types (use isRetriable())
  • Check if errors are non-retriable (400, 401, 404)
  • Increase delays between retries
  • Reduce concurrent requests to avoid overwhelming targets
  • Check account balance (insufficient balance causes failures)

Need Help?