← Blog

Connect HProxy to CloakBrowser and reduce unnecessary downloads

A catalog task may need product names, prices and availability while the browser also downloads every product photograph. With a residential proxy billed by the gigabyte, that extra work can consume traffic without adding useful records. CloakBrowser lets you test which downloads your task actually needs.

Quick answer: Connect a generated HProxy HTTP proxy to CloakBrowser, verify the browser’s exit IP, then run the same catalog task twice. First allow all resources; then block images and media. Keep the change only if the required records still match and the measured traffic falls. JavaScript and data requests stay enabled.

We tested the example with real CloakBrowser against a synthetic catalog through a local authenticated proxy. The HProxy setup below follows its documentation and supplied guide. We did not test HProxy’s residential network or measure an HProxy bill.

Connect one HProxy HTTP line

Start with an active HProxy residential plan and a working CloakBrowser installation. The CloakBrowser Quick Start covers the first browser launch. Complete license setup before running the comparison; keep keys out of your script.

python -m pip install --upgrade cloakbrowser

Open your residential plan in HProxy’s dashboard and use its generator to create an HTTP line with username/password authentication. For a catalog that varies by location, select one country and a sticky session supported by your plan. Reuse the same generated line throughout the comparison. A sticky session reduces one source of variation; it does not guarantee a permanent address or identical page content.

HProxy’s line-generation documentation describes the result as host:port:username:password. Keep the complete generated username, including targeting/session parameters. Use the proxy credentials, not your dashboard login. The script asks for the fields separately:

Generator fieldInput to the script
HostHostname only, without a URL scheme
PortThe port generated for HTTP
UsernameComplete proxy username, unchanged
PasswordProxy password; hidden when entered

The code builds server as http://HOST:PORT. It also passes username and password as separate fields, so you do not need to percent-encode credentials into a URL. HTTP here describes the proxy connection; HTTPS destinations are still supported. Do not convert a SOCKS5 line by changing only its prefix.

Build a comparison that checks the data

Save the next three code blocks, in order, in one file named hproxy_compare.py. They form one complete script. It uses the ordinary Playwright API exposed by CloakBrowser; our run used CloakBrowser 0.7.1 and Playwright 1.62.0.

The example expects a catalog you own or are allowed to test. In our fixture, JavaScript loads a JSON response and creates product elements with data-product-id, data-price, data-stock and a child named .name. The container becomes #catalog[data-ready="true"] when the data is ready. Adapt these selectors and the readiness condition to your page; they are an example contract, not universal selectors. Supply a known expected record count and check sample records against the source.

import json
import warnings
from getpass import GetPassWarning, getpass
from ipaddress import ip_address
from time import monotonic, sleep
from urllib.parse import urlsplit

from cloakbrowser import launch


def read_products(page, expected_count):
    rows = page.locator('[data-product-id]').evaluate_all('''els =>
        els.map(e => ({
            id: e.dataset.productId,
            name: e.querySelector('.name')?.textContent.trim(),
            price: e.dataset.price,
            stock: e.dataset.stock
        }))''')
    if (len(rows) != expected_count
            or len({r['id'] for r in rows}) != expected_count
            or any(not all(r.values()) for r in rows)):
        raise ValueError('Missing, duplicate or incomplete products')
    return sorted(rows, key=lambda r: r['id'])

The data check requires the expected number of distinct product IDs and nonempty fields, then sorts records by ID so a harmless display-order change does not count as a different result. Use a stable catalog for the experiment: a genuine price update between requests makes the comparison inconclusive.

The optimization is intentionally narrow. Images and media are candidates for removal when the output is structured text data. Keep JavaScript and data requests enabled; they build the records in this example. A page that needs its images to complete the task should keep those images too.

Measure one load

Append this function. It records completed HTTP requests, tracks intentional blocks separately from unexpected failures, and always closes its browser context. The full and selective modes use the same cache and service-worker settings.

def measure(browser, url, expected_count, trim):
    context = browser.new_context(service_workers='block')
    pending, blocked, failures, sizes = set(), set(), [], []

    def route_request(route):
        request = route.request
        if trim and request.resource_type in {'image', 'media'}:
            blocked.add(request)
            route.abort()
        else:
            route.continue_()

    def finished(request):
        sizes.append(sum(request.sizes().values()))
        pending.discard(request)

    def failed(request):
        pending.discard(request)
        if request not in blocked:
            failures.append('network')

    try:
        # Routing in both modes keeps HTTP caching equivalent.
        context.route('**/*', route_request)
        page = context.new_page()
        page.on('request', lambda request: pending.add(request))
        page.on('requestfinished', finished)
        page.on('requestfailed', failed)
        page.on('response', lambda r:
                failures.append('http') if r.status >= 400 else None)
        start = monotonic()
        response = page.goto(url, wait_until='load', timeout=30000)
        if response is None or response.status != 200:
            raise ValueError('Page did not return HTTP 200')
        page.locator('#catalog[data-ready="true"]').wait_for()
        # Fixed observation window, identical for both modes.
        sleep(1)
        rows = read_products(page, expected_count)
        if pending or failures:
            raise ValueError('Incomplete load or unexpected request errors')
        return {'products': rows, 'http_bytes': sum(sizes),
                'completed': len(sizes), 'blocked': len(blocked),
                'seconds': round(monotonic() - start, 3)}
    finally:
        context.close()

Add the connection and comparison entry point

Append this block to the same file. The initial IP check runs through a browser page, then closes that context before measurement. The two catalog runs use fresh contexts and share the same proxy configuration.

def compare(browser, url, expected_count):
    full = measure(browser, url, expected_count, False)
    lean = measure(browser, url, expected_count, True)
    if full['products'] != lean['products']:
        raise ValueError('Product data changed; reject this optimization')
    if full['http_bytes'] <= 0:
        raise ValueError('No baseline bytes recorded')
    reduction = 100 * (1 - lean['http_bytes'] / full['http_bytes'])
    return {'full': full, 'lean': lean, 'data_equal': True,
            'http_byte_reduction_percent': round(reduction, 2)}


def main():
    host = input('HTTP proxy host (no scheme): ').strip()
    port = int(input('HTTP proxy port: '))
    user = input('Proxy username: ').strip()
    with warnings.catch_warnings():
        warnings.simplefilter('error', GetPassWarning)
        password = getpass('Proxy password: ')
    url = input('Your catalog URL: ').strip()
    count = int(input('Expected product count: '))
    if (not host or any(c in host for c in '/:@?# ') or not user
            or not password or not 1 <= port <= 65535 or count < 1
            or urlsplit(url).scheme not in {'http', 'https'}
            or not urlsplit(url).hostname):
        raise ValueError('Check the connection and catalog inputs')
    proxy = {'server': f'http://{host}:{port}',
             'username': user, 'password': password}
    browser = launch(proxy=proxy)
    try:
        page = browser.new_page()
        try:
            response = page.goto('https://api.ipify.org?format=json')
            if response is None or response.status != 200:
                raise ValueError('IP check failed')
            data = json.loads(page.locator('body').inner_text())
            print('Browser exit IP:', ip_address(str(data['ip'])))
        finally:
            page.context.close()
        print(json.dumps(compare(browser, url, count), indent=2))
    finally:
        browser.close()


if __name__ == '__main__':
    try:
        main()
    except Exception as exc:
        # Raw connection errors can contain credentials or private URLs.
        print('Check failed:', type(exc).__name__)
        raise SystemExit(1) from None

Run it and inspect both results

python hproxy_compare.py

Enter the four proxy fields, your reachable catalog URL and its expected product count. A remote HProxy exit cannot reach a catalog running only on your laptop’s localhost. Use an authorized staging URL for your own HProxy run; our internal local test used a local forwarding proxy.

The first line prints the exit IP seen by ipify’s JSON endpoint. Compare it with the exit you expect, not the gateway hostname. The following JSON contains full and lean results, the extracted records, completed and blocked request counts, HTTP byte totals and the percentage difference. A valid IP check establishes this route to ipify, not access to every target.

The selective mode blocks only Playwright resource types image and media, which includes video and audio. It leaves scripts, stylesheets, fonts, documents, XHR and fetch requests alone. Blocking by resource type covers images without file extensions too; image data fetched through an ordinary fetch request will remain allowed.

What our controlled test showed

Our synthetic catalog contained 12 products, 12 generated 640 x 480 JPEGs and one short WebM video. JavaScript fetched the product JSON and populated the page. We ran three full/lean pairs through an authenticated local HTTP proxy, alternating their order. Every run returned exactly the expected IDs, names, prices and stock values.

MeasurementFull loadSelective load
Correct product records1212, identical
Completed requests174
Requests deliberately blocked013
Browser-observed HTTP bytes788,0853,544
Proxy-observed HTTP response bytes783,6912,496

Each of the three pairs produced these same byte totals: 99.55% fewer browser-observed HTTP bytes on this fixture. The local proxy independently confirmed that no JPEG or video request reached the origin in selective mode. A separate full load with no routing handler returned the same records and proxy response-byte total as the full comparison mode.

This large difference reflects a deliberately image-heavy test catalog. It is not an expected saving for your website, an HProxy performance benchmark or a billing discount. The two byte counters also measure different things: the browser total includes request and response headers/bodies, while the proxy figure above counts HTTP responses only.

Decide whether the change belongs in your task

Correctness comes first. The script rejects missing records, duplicate IDs, empty fields, differing results, unfinished requests and unexpected network/HTTP errors. Our checks also rejected an incorrect expected count, a broken JSON load and an explicitly injected price change with the record count unchanged. Matching counts alone would have missed that last case.

Keep the comparison fair. Playwright disables HTTP caching when routing is enabled. Both modes therefore install a handler: full mode allows everything, lean mode blocks selected types. Both block service workers so they cannot bypass interception. This is a comparison of fresh contexts; it does not predict the benefit on a warm, cached returning visit. Apps that require a service worker need a different test setup.

Use a defined observation window. This example waits for page load, catalog readiness and one additional second. It rejects requests still in flight when it reads the result. For lazy loading, pagination, long polling or streaming, replace that boundary with your task’s explicit completion condition and measure the same work in both modes. Do not remove error checks merely to obtain a smaller number.

Separate page measurements from billing. The byte total uses Playwright’s request size information for completed page requests. It excludes connection/TLS overhead, background browser traffic, the initial IP check and downloads outside the measurement window. It is not a full network meter. Failed or deliberately aborted transfers are not included as completed requests.

For HProxy usage, compare equivalent batches on an otherwise idle plan and consult the provider’s meter. HProxy documents traffic statistics for Residential Premium and a timestamp on individual-plan usage readings; the general plan list can lag. Do not treat a tiny immediate balance difference, or traffic from a shared plan, as a precise cost measurement. More generated proxy lines still draw from the same plan balance.

When should images stay enabled?

Keep them for screenshots, visual QA, image extraction or a page whose behavior depends on image loading. A catalog can also depend on image load events to reveal data or controls. If selective loading changes the result, narrow the block rule to resources you have verified as unnecessary, or keep the full load.

Does a different price prove that blocking broke the page?

No. Live prices, cookies, location or experiments can change content between visits. The script rejects the comparison because it is inconclusive; it does not diagnose the cause. Use a stable fixture or controlled staging data to isolate the effect, then repeat on representative authorized pages.

What if the script reports a failed check?

For a connection error, verify the generated HTTP port and proxy credentials first. For a timeout or data error, check your selectors, readiness condition, expected records and required requests. The command prints only the exception class to avoid exposing credentials in raw connection errors. Keep any deeper diagnostics private and sanitize them before sharing.

Put the validated change to work

Keep the full and selective results as a regression check when your target changes. Measure bytes per correct result over representative batches, including retries, before choosing a production setting. Our guide to browser automation in production explains why a completed run can still return unusable data.

Connect HProxy to CloakBrowser, start with one working catalog task and test which downloads it can safely omit. If you are setting up the browser for the first time, begin with the CloakBrowser Quick Start.

Partner offer: CLOAKBROWSER20 gives 20% off HProxy Residential Premium, once per customer, for new and existing customers, through March 31, 2027.

Disclosure: CloakBrowser does not receive a commission from HProxy. The CLOAKBROWSER20 coupon provides a discount for readers only.