How To Extract Text With Formatting Using Beautifulsoup

Overview: Extract formatted text

This guide shows you how to extract formatted text using BeautifulSoup. We cover installation, practical code examples, proxy configuration for production, and error handling best practices.

Prerequisites & Installation

pip install beautifulsoup4 requests lxml selenium playwright
  # Install Playwright browsers:
  playwright install chromium

Complete Working Code Example

import requests
  from bs4 import BeautifulSoup

  # Configure rotating proxy (Cheapest Proxies - $0.99/GB)
  proxy = {
      'http': 'http://USER:PASS@proxy.cheapest-proxies.com:8000',
      'https': 'http://USER:PASS@proxy.cheapest-proxies.com:8000'
  }

  headers = {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
      'Accept-Language': 'en-US,en;q=0.9',
  }

  response = requests.get('https://example.com', proxies=proxy, headers=headers, timeout=30)
  soup = BeautifulSoup(response.text, 'lxml')

  # Extract formatted text
  elements = soup.find_all('div', class_='target')
  for el in elements:
      print(el.get_text(strip=True))

Step-by-Step Explanation

Adding Proxies for Production Scale

def scrape_with_retry(url, max_retries=3):
      proxy = {
          'http': 'http://USER:PASS@proxy.cheapest-proxies.com:8000',
          'https': 'http://USER:PASS@proxy.cheapest-proxies.com:8000'
      }
      for attempt in range(max_retries):
          try:
              r = requests.get(url, proxies=proxy, timeout=30)
              r.raise_for_status()
              return BeautifulSoup(r.text, 'lxml')
          except Exception as e:
              print(f"Retry {attempt+1}: {e}")
      return None

Common Variations

elements = soup.select("div.product > span.price")

tree = html.fromstring(response.content)
  prices = tree.xpath('//span[@class="price"]/text()')

el = soup.find('div', {'class':'x', 'id':'y'})

el = soup.find('div', class_='x')
  child = el.find('p')

Error Handling Best Practices

from bs4 import BeautifulSoup
  import requests

  def safe_scrape(url, proxy):
      try:
          r = requests.get(url, proxies=proxy, timeout=30)
          r.raise_for_status()
          soup = BeautifulSoup(r.text, 'lxml')
          el = soup.find('div', class_='target')
          return el.get_text(strip=True) if el else None
      except requests.exceptions.ProxyError:
          print("Proxy failed — check credentials")
      except requests.exceptions.Timeout:
          print("Request timed out")
      except requests.exceptions.HTTPError as e:
          print(f"HTTP error: {e.response.status_code}")
      return None

Best Practices

FAQ