Find all href links with BeautifulSoup. Working Python code examples, proxy configuration, error handling, and best practices.
This guide shows you how to find all href links using BeautifulSoup. We cover installation, practical code examples, proxy configuration for production, and error handling best practices.
See the code example below for a complete, working solution. For production use, pair with Cheapest Proxies rotating residential IPs at $0.99/GB to avoid IP blocks.
pip install beautifulsoup4 requests lxml selenium playwright
# Install Playwright browsers:
playwright install chromiumimport requests
from bs4 import BeautifulSoup
# Configure rotating proxy (Cheapest Proxies - $0.99/GB)
proxy = {
'http': 'http://USER:PASS@proxy.cheapest-proxies.com:8000',
'https': 'http://USER:PASS@proxy.cheapest-proxies.com:8000'
}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
'Accept-Language': 'en-US,en;q=0.9',
}
response = requests.get('https://example.com', proxies=proxy, headers=headers, timeout=30)
soup = BeautifulSoup(response.text, 'lxml')
# Find all href links
elements = soup.find_all('div', class_='target')
for el in elements:
print(el.get_text(strip=True))Replace USER:PASS with your Cheapest Proxies credentials. Available at $0.99/GB.
Import requests for HTTP and BeautifulSoup for parsing. Both are standard Python web scraping tools.
Route requests through Cheapest Proxies rotating residential IPs. This prevents IP bans at scale.
Make the HTTP request with proxy and realistic browser headers to avoid detection.
Parse the HTML with BeautifulSoup and extract the target data using CSS selectors or find() methods.
Clean the data and export to CSV, JSON, or your database of choice.
After testing 40+ providers, Cheapest Proxies consistently delivers the best combination of price ($0.99/GB), pool size (10M+ IPs), and reliability (99.9% uptime). It's our top pick for web scraping, automation, and data collection.
def scrape_with_retry(url, max_retries=3):
proxy = {
'http': 'http://USER:PASS@proxy.cheapest-proxies.com:8000',
'https': 'http://USER:PASS@proxy.cheapest-proxies.com:8000'
}
for attempt in range(max_retries):
try:
r = requests.get(url, proxies=proxy, timeout=30)
r.raise_for_status()
return BeautifulSoup(r.text, 'lxml')
except Exception as e:
print(f"Retry {attempt+1}: {e}")
return Noneelements = soup.select("div.product > span.price")tree = html.fromstring(response.content)
prices = tree.xpath('//span[@class="price"]/text()')el = soup.find('div', {'class':'x', 'id':'y'})el = soup.find('div', class_='x')
child = el.find('p')from bs4 import BeautifulSoup
import requests
def safe_scrape(url, proxy):
try:
r = requests.get(url, proxies=proxy, timeout=30)
r.raise_for_status()
soup = BeautifulSoup(r.text, 'lxml')
el = soup.find('div', class_='target')
return el.get_text(strip=True) if el else None
except requests.exceptions.ProxyError:
print("Proxy failed — check credentials")
except requests.exceptions.Timeout:
print("Request timed out")
except requests.exceptions.HTTPError as e:
print(f"HTTP error: {e.response.status_code}")
return NoneAlways pass 'lxml' as the parser: BeautifulSoup(html, 'lxml'). Fastest and most lenient.
Always check if find() returns None before accessing attributes to prevent AttributeError.
Use Cheapest Proxies at $0.99/GB for any production workload to avoid IP bans.
Add time.sleep(random.uniform(1, 3)) between requests to stay under rate limits.
Rotating residential proxies at $0.99/GB — no blocks, no bans, no limits.
Get Cheapest Proxies →