Error Handling Guide
Best practices for handling errors and implementing robust retry logic.
Overview
Web scraping is inherently prone to errors. Websites can be down, networks can fail, CAPTCHAs can appear unexpectedly, and dynamic content can take too long to load. Robust error handling is essential for building reliable scraping systems that can recover from failures and continue operating.
Browser7 provides detailed error information and status codes to help you diagnose and handle errors effectively. This guide covers common error types, best practices for error handling, and strategies for implementing retry logic.
Why Error Handling Matters
Production Reliability
In production systems, errors are inevitable:
- Network Failures: Temporary connectivity issues
- Website Changes: Page structure changes breaking selectors
- Rate Limiting: Too many requests in a short time
- Service Outages: Target websites or Browser7 temporarily unavailable
Without proper error handling, a single failure can crash your entire scraping job.
Cost Optimization
Not all failed renders are charged. User-caused failures (invalid selectors, target site errors, timeouts) are charged $0.01, but system errors (Browser7 infrastructure failures) are never charged. Check the billable field on failed renders to verify whether a charge was applied. See Error Codes Reference for the complete breakdown.
- Detect Failures Early: Check status and
errorCodebefore processing HTML - Implement Smart Retries: Don't retry unrecoverable errors; retry system errors immediately
- Monitor Error Rates: Identify problematic sites and adjust strategies
- Log Error Details: Understand why renders fail to improve success rates
Data Quality
Errors can result in incomplete or incorrect data:
- Partial Renders: HTML captured before content loads
- Missing Elements: Selectors don't match expected content
- Stale Data: Cached responses instead of fresh data
- CAPTCHA Blocks: Protected content not accessible
Proper error handling ensures you only process complete, accurate data.
Common Error Types
Network Errors
Connection failures before the render starts:
Symptoms:
'Failed to connect to'in error message- Network timeouts
- DNS resolution failures
Causes:
- Poor network connectivity
- Firewall or proxy issues
- Browser7 API temporarily unreachable
Handling:
import { Browser7Error } from 'browser7';
try {
const result = await client.render(url);
} catch (error) {
if (error instanceof Browser7Error && !error.statusCode) {
console.error('Network error - retrying in 5 seconds...');
await delay(5000);
// Retry logic here
}
}
API Errors
Browser7 API rejects the request before rendering:
Symptoms:
'Failed to start render'in error message- HTTP status codes (400, 401, 403, 429, 500)
- Invalid parameters or authentication
Common causes:
- 400: Invalid request parameters (bad URL, invalid country code, etc.)
- 401: Invalid or missing API key
- 403: Insufficient permissions or account suspended
- 429: Rate limit exceeded
- 500: Server error (temporary)
Handling:
import { AuthenticationError, RateLimitError, ValidationError } from 'browser7';
try {
const result = await client.render(url, options);
} catch (error) {
if (error instanceof AuthenticationError) {
console.error('Invalid API key');
// Don't retry - fix authentication
} else if (error instanceof RateLimitError) {
console.error('Rate limited - backing off...');
await delay(60000);
// Retry with backoff
} else if (error instanceof ValidationError) {
console.error('Invalid parameters:', error.details);
// Don't retry - fix parameters
}
}
Render Failures
Render starts but fails during execution:
Symptoms:
status: 'failed'in resulterrorfield contains failure reason
Common causes:
- Target website unreachable or down
- Invalid URL (404, 500 errors from target site)
- Timeout during page load
- Browser crashes or resource exhaustion
Handling:
const result = await client.render(url);
if (result.status === 'failed') {
console.error('Render failed:', result.error);
// Check if retriable
if (result.error.includes('timeout')) {
// Retry with longer timeout or wait actions
} else if (result.error.includes('404')) {
// Don't retry - URL doesn't exist
}
}
Timeout Errors
Render exceeds maximum polling attempts:
Symptoms:
'timed out after 60 attempts'in error message- Very long render times (>2 minutes)
Causes:
- Extremely slow target website
- Complex JavaScript taking too long to execute
- Network issues delaying responses
- Browser7 processing queue backup
Handling:
try {
const result = await client.render(url);
} catch (error) {
if (error.message.includes('timed out')) {
console.error('Render timed out - site may be too slow');
// Consider simplifying wait actions or using different approach
}
}
Wait Action Failures
Wait actions timeout or fail:
Symptoms:
result.timingBreakdown.waitActionsMsis very high (close to timeout)- Content missing from HTML
- Status is 'completed' but data is incomplete
Causes:
- Selectors don't match any elements
- Elements never become visible
- Text content never appears
- Timeouts too short for slow-loading content
Handling:
const result = await client.render(url, { waitFor: actions });
if (result.timingBreakdown.waitActionsMs > 25000) {
console.warn('Wait actions took unusually long');
// Check if expected content exists in HTML
if (!result.html.includes('expected-content')) {
// Adjust wait actions or retry with different strategy
}
}
CAPTCHA Failures
CAPTCHA detected but not solved:
Symptoms:
result.captcha.detected: trueresult.captcha.handled: false- HTML contains CAPTCHA widget
Causes:
- CAPTCHA type not supported
- CAPTCHA solving service temporarily unavailable
- Complex CAPTCHA configuration
- Site requires additional verification
Handling:
const result = await client.render(url, { captcha: 'auto' });
if (result.captcha.detected && !result.captcha.handled) {
console.error('CAPTCHA detected but not solved');
console.log('CAPTCHA type:', result.captcha.sitekey);
// Retry with specific CAPTCHA type or contact support
}
Best Practices
Always Use Try-Catch
Wrap all render calls in try-catch blocks:
async function safeFetch(url) {
try {
const result = await client.render(url);
return result;
} catch (error) {
console.error(`Error fetching ${url}:`, error.message);
return null;
}
}
Check Status Before Processing
Always verify status: 'completed' before using HTML:
const result = await client.render(url);
if (result.status !== 'completed') {
console.error('Render did not complete:', result.status);
if (result.error) {
console.error('Error:', result.error);
}
return null;
}
// Safe to process HTML
processHTML(result.html);
Implement Exponential Backoff
Retry with increasing delays between attempts:
async function renderWithRetry(url, maxRetries = 3) {
let lastError;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const result = await client.render(url);
if (result.status === 'completed') {
return result;
}
// Render failed, but no exception thrown
lastError = result.error;
} catch (error) {
lastError = error.message;
}
if (attempt < maxRetries) {
// Exponential backoff: 2s, 4s, 8s, etc.
const delayMs = Math.pow(2, attempt) * 1000;
console.log(`Retry ${attempt}/${maxRetries} in ${delayMs}ms...`);
await new Promise(resolve => setTimeout(resolve, delayMs));
}
}
throw new Error(`Failed after ${maxRetries} attempts: ${lastError}`);
}
Distinguish Retriable vs Non-Retriable Errors
Don't retry errors that won't succeed:
import { AuthenticationError, ValidationError, RateLimitError, Browser7Error } from 'browser7';
function isRetriable(error) {
// Non-retriable: authentication, validation, not found
if (error instanceof AuthenticationError) return false;
if (error instanceof ValidationError) return false;
if (error.statusCode === 404) return false;
// Retriable: rate limits (with delay), server errors, network errors
if (error instanceof RateLimitError) return true;
if (error.statusCode >= 500) return true;
if (error instanceof Browser7Error && !error.statusCode) return true;
// Unknown errors - retry cautiously
return true;
}
async function smartRetry(url, maxRetries = 3) {
let lastError;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
return await client.render(url);
} catch (error) {
lastError = error;
if (!isRetriable(error)) {
console.error('Non-retriable error:', error.message);
throw error; // Don't retry
}
if (attempt < maxRetries) {
await delay(Math.pow(2, attempt) * 1000);
}
}
}
throw lastError;
}
Log Comprehensive Error Details
Include context for debugging:
try {
const result = await client.render(url, options);
if (result.status === 'failed') {
console.error('Render failed:', {
url,
error: result.error,
selectedCity: result.selectedCity?.displayName,
timingBreakdown: result.timingBreakdown,
captcha: result.captcha
});
}
} catch (error) {
console.error('Exception during render:', {
url,
options,
error: error.message,
stack: error.stack
});
}
Set Reasonable Timeouts
Balance reliability with speed:
// Fast timeouts for quick fails (development)
Browser7.waitForSelector('.content', 'visible', 5000)
// Longer timeouts for reliability (production)
Browser7.waitForSelector('.content', 'visible', 30000)
// Very long for known slow sites
Browser7.waitForSelector('.content', 'visible', 60000)
Monitor Error Rates
Track success and failure rates:
const stats = {
total: 0,
success: 0,
failed: 0,
retried: 0,
errors: {}
};
async function monitoredRender(url) {
stats.total++;
try {
const result = await client.render(url);
if (result.status === 'completed') {
stats.success++;
return result;
} else {
stats.failed++;
stats.errors[result.error] = (stats.errors[result.error] || 0) + 1;
return null;
}
} catch (error) {
stats.failed++;
const errorKey = error.message.substring(0, 50);
stats.errors[errorKey] = (stats.errors[errorKey] || 0) + 1;
throw error;
}
}
// Periodic reporting
setInterval(() => {
console.log('Success rate:', (stats.success / stats.total * 100).toFixed(2) + '%');
console.log('Top errors:', Object.entries(stats.errors).sort((a, b) => b[1] - a[1]).slice(0, 5));
}, 60000);
Implement Circuit Breakers
Stop attempting when a site consistently fails:
class CircuitBreaker {
constructor(threshold = 5, timeout = 60000) {
this.failures = 0;
this.threshold = threshold;
this.timeout = timeout;
this.state = 'closed'; // closed, open, half-open
this.nextAttempt = null;
}
async execute(fn) {
if (this.state === 'open') {
if (Date.now() < this.nextAttempt) {
throw new Error('Circuit breaker is open');
}
this.state = 'half-open';
}
try {
const result = await fn();
this.onSuccess();
return result;
} catch (error) {
this.onFailure();
throw error;
}
}
onSuccess() {
this.failures = 0;
this.state = 'closed';
}
onFailure() {
this.failures++;
if (this.failures >= this.threshold) {
this.state = 'open';
this.nextAttempt = Date.now() + this.timeout;
console.error('Circuit breaker opened');
}
}
}
const breaker = new CircuitBreaker(5, 60000);
async function protectedRender(url) {
return breaker.execute(() => client.render(url));
}
Handling Specific Errors
Rate Limiting (429)
Respect rate limits with backoff:
async function handleRateLimit(url, retries = 3) {
for (let i = 0; i < retries; i++) {
try {
return await client.render(url);
} catch (error) {
if (error.message.includes('429')) {
const delayMs = Math.pow(2, i + 5) * 1000; // 32s, 64s, 128s
console.log(`Rate limited - waiting ${delayMs}ms...`);
await delay(delayMs);
} else {
throw error;
}
}
}
throw new Error('Rate limit persists after retries');
}
Authentication Errors (401)
Verify and refresh credentials:
async function renderWithAuth(url) {
try {
return await client.render(url);
} catch (error) {
if (error.message.includes('401')) {
console.error('Authentication failed - check API key');
// Fetch fresh API key from secure storage
// Create new client
throw new Error('Invalid API credentials');
}
throw error;
}
}
Target Site Errors (404, 500)
Handle target website issues:
const result = await client.render(url);
if (result.status === 'failed') {
if (result.error.includes('404')) {
console.error('URL not found:', url);
// Mark URL as invalid, don't retry
await markUrlInvalid(url);
return null;
} else if (result.error.includes('500')) {
console.error('Target site error - retrying...');
// Target site issue, retry after delay
await delay(10000);
return renderWithRetry(url);
}
}
Wait Action Timeouts
Adjust wait actions based on failures:
async function adaptiveRender(url, options) {
try {
const result = await client.render(url, options);
if (result.timingBreakdown.waitActionsMs > 25000) {
console.warn('Wait actions slow - adjusting timeouts');
// Increase timeouts for next attempt
options.waitFor = options.waitFor.map(action => ({
...action,
timeout: (action.timeout || 30000) * 1.5
}));
}
return result;
} catch (error) {
if (error.message.includes('timed out')) {
// Simplify wait actions
options.waitFor = options.waitFor.slice(0, 2);
return client.render(url, options);
}
throw error;
}
}
Retry Strategies
Simple Retry
Basic retry with fixed delay:
async function simpleRetry(url, attempts = 3, delayMs = 2000) {
for (let i = 0; i < attempts; i++) {
try {
return await client.render(url);
} catch (error) {
if (i === attempts - 1) throw error;
await delay(delayMs);
}
}
}
Exponential Backoff
Increasing delays between retries:
async function exponentialBackoff(url, maxAttempts = 5) {
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await client.render(url);
} catch (error) {
if (attempt === maxAttempts - 1) throw error;
const delayMs = Math.min(
Math.pow(2, attempt) * 1000,
32000 // Max 32 seconds
);
console.log(`Attempt ${attempt + 1} failed, retrying in ${delayMs}ms`);
await delay(delayMs);
}
}
}
Jittered Backoff
Add randomness to prevent thundering herd:
async function jitteredBackoff(url, maxAttempts = 5) {
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await client.render(url);
} catch (error) {
if (attempt === maxAttempts - 1) throw error;
const baseDelay = Math.pow(2, attempt) * 1000;
const jitter = Math.random() * 1000;
const delayMs = Math.min(baseDelay + jitter, 32000);
await delay(delayMs);
}
}
}
Conditional Retry
Only retry specific error types:
async function conditionalRetry(url, maxAttempts = 3) {
const retriableErrors = ['timeout', 'ECONNREFUSED', '500', '502', '503'];
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
const result = await client.render(url);
if (result.status === 'completed') {
return result;
}
// Check if error is retriable
const isRetriable = retriableErrors.some(err =>
result.error?.includes(err)
);
if (!isRetriable || attempt === maxAttempts - 1) {
throw new Error(result.error);
}
await delay(Math.pow(2, attempt) * 1000);
} catch (error) {
const isRetriable = retriableErrors.some(err =>
error.message.includes(err)
);
if (!isRetriable || attempt === maxAttempts - 1) {
throw error;
}
await delay(Math.pow(2, attempt) * 1000);
}
}
}
Common Use Cases
Robust Scraper with Full Error Handling
async function robustScraper(urls) {
const results = [];
const errors = [];
for (const url of urls) {
try {
const result = await exponentialBackoff(url, 3);
if (result.status === 'completed') {
results.push({
url,
html: result.html,
city: result.selectedCity.displayName,
timing: result.timingBreakdown.totalMs
});
} else {
errors.push({ url, error: result.error });
}
} catch (error) {
errors.push({ url, error: error.message });
}
}
console.log(`Success: ${results.length}/${urls.length}`);
console.log(`Errors: ${errors.length}/${urls.length}`);
return { results, errors };
}
Production-Grade Error Handler
class ProductionRenderer {
constructor(client, options = {}) {
this.client = client;
this.maxRetries = options.maxRetries || 3;
this.breaker = new CircuitBreaker();
this.stats = { success: 0, failed: 0, retried: 0 };
}
async render(url, options = {}) {
return this.breaker.execute(async () => {
let lastError;
for (let attempt = 0; attempt < this.maxRetries; attempt++) {
try {
const result = await this.client.render(url, options);
if (result.status === 'completed') {
this.stats.success++;
if (attempt > 0) this.stats.retried++;
return result;
}
lastError = result.error;
if (!this.isRetriable(lastError)) {
throw new Error(lastError);
}
} catch (error) {
lastError = error.message;
if (!this.isRetriable(lastError)) {
this.stats.failed++;
throw error;
}
}
if (attempt < this.maxRetries - 1) {
await this.delay(Math.pow(2, attempt) * 1000);
}
}
this.stats.failed++;
throw new Error(`Failed after ${this.maxRetries} attempts: ${lastError}`);
});
}
isRetriable(error) {
const nonRetriable = ['400', '401', '403', '404'];
return !nonRetriable.some(code => error.includes(code));
}
async delay(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
getStats() {
return this.stats;
}
}
SDK Implementations
See language-specific examples for error handling:
- Node.js Examples - Error handling with the Node.js SDK
Python Examples
import time
from browser7 import Browser7
client = Browser7(api_key='b7_your_api_key_here')
# Basic try/except
try:
result = client.render('https://example.com')
print(result.html)
except Exception as e:
print(f'Error: {e}')
# Check status before processing
result = client.render('https://example.com')
if result.status != 'completed':
print(f'Render did not complete: {result.status}')
if result.error:
print(f'Error: {result.error}')
else:
process_html(result.html)
# Retry with exponential backoff
def render_with_retry(url, max_retries=3):
last_error = None
for attempt in range(1, max_retries + 1):
try:
result = client.render(url)
if result.status == 'completed':
return result
last_error = result.error
except Exception as e:
last_error = str(e)
if attempt < max_retries:
delay = (2 ** attempt) # 2s, 4s, 8s
print(f'Retry {attempt}/{max_retries} in {delay}s...')
time.sleep(delay)
raise Exception(f'Failed after {max_retries} attempts: {last_error}')
# Distinguish retriable vs non-retriable errors
NON_RETRIABLE = ['401', '400', '403', '404']
def is_retriable(error):
return not any(code in str(error) for code in NON_RETRIABLE)
def smart_retry(url, max_retries=3):
last_error = None
for attempt in range(max_retries):
try:
result = client.render(url)
if result.status == 'completed':
return result
last_error = result.error
if not is_retriable(last_error):
raise Exception(last_error)
except Exception as e:
if not is_retriable(str(e)):
raise
last_error = str(e)
if attempt < max_retries - 1:
time.sleep(2 ** (attempt + 1))
raise Exception(f'Failed after {max_retries} attempts: {last_error}')
PHP Examples
<?php
use Browser7\Browser7Client;
$client = new Browser7Client('b7_your_api_key_here');
// Basic try/catch
try {
$result = $client->render('https://example.com');
echo $result->html;
} catch (\RuntimeException $e) {
echo 'Error: ' . $e->getMessage();
}
// Check status before processing
$result = $client->render('https://example.com');
if ($result->status !== 'completed') {
echo 'Render did not complete: ' . $result->status;
if ($result->error) {
echo 'Error: ' . $result->error;
}
} else {
processHtml($result->html);
}
// Retry with exponential backoff
function renderWithRetry(Browser7Client $client, string $url, int $maxRetries = 3): mixed
{
$lastError = null;
for ($attempt = 1; $attempt <= $maxRetries; $attempt++) {
try {
$result = $client->render($url);
if ($result->status === 'completed') {
return $result;
}
$lastError = $result->error;
} catch (\RuntimeException $e) {
$lastError = $e->getMessage();
}
if ($attempt < $maxRetries) {
$delay = pow(2, $attempt); // 2s, 4s, 8s
echo "Retry {$attempt}/{$maxRetries} in {$delay}s...\n";
sleep($delay);
}
}
throw new \RuntimeException("Failed after {$maxRetries} attempts: {$lastError}");
}
// Distinguish retriable vs non-retriable errors
function isRetriable(string $error): bool
{
$nonRetriable = ['400', '401', '403', '404'];
foreach ($nonRetriable as $code) {
if (str_contains($error, $code)) {
return false;
}
}
return true;
}
function smartRetry(Browser7Client $client, string $url, int $maxRetries = 3): mixed
{
$lastError = null;
for ($attempt = 0; $attempt < $maxRetries; $attempt++) {
try {
$result = $client->render($url);
if ($result->status === 'completed') {
return $result;
}
$lastError = $result->error;
if (!isRetriable($lastError)) {
throw new \RuntimeException($lastError);
}
} catch (\RuntimeException $e) {
if (!isRetriable($e->getMessage())) {
throw $e;
}
$lastError = $e->getMessage();
}
if ($attempt < $maxRetries - 1) {
sleep(pow(2, $attempt + 1));
}
}
throw new \RuntimeException("Failed after {$maxRetries} attempts: {$lastError}");
}
Related Guides
- Wait Actions - Handle wait action failures
- CAPTCHA Solving - Handle CAPTCHA failures
- Geo-Targeting - Handle geo-targeting errors
Troubleshooting
High Error Rates
If you're experiencing frequent errors:
- Check
result.errormessages to identify patterns - Monitor
result.timingBreakdownfor timeouts - Verify target websites are accessible independently
- Check Browser7 status page for service issues
- Review rate limiting and adjust request frequency
- Simplify wait actions to reduce timeouts
Renders Always Fail for Specific Sites
If certain sites consistently fail:
- Test site accessibility in a regular browser
- Check if site has anti-bot protections (Cloudflare, etc.)
- Try different geo-targeting locations
- Enable CAPTCHA solving if not already enabled
- Simplify or remove wait actions
- Contact support with specific site details
Intermittent Failures
If errors occur randomly:
- Implement retry logic with exponential backoff
- Check network connectivity and stability
- Monitor target site reliability (some sites are unreliable)
- Add circuit breakers to prevent cascade failures
- Increase wait action timeouts for reliability
Retries Not Helping
If retries don't improve success rates:
- Verify you're retrying the right error types (use
isRetriable()) - Check if errors are non-retriable (400, 401, 404)
- Increase delays between retries
- Reduce concurrent requests to avoid overwhelming targets
- Check account balance (insufficient balance causes failures)
Need Help?
- 📚 API Reference - Complete error response documentation
- 💬 Support - Contact our team
- 📊 Status Page - Check service status
- 🐛 Report Issues - SDK-specific issues