What problems is Bright Data solving and how is that benefiting you?
Our research team has developed a tool that mimics end users to automate queries of ISP interfaces and extract advertised broadband availability, speed tiers, and pricing data at street-address granularity. To avoid originating all queries from a single non-residential IP address, we leverage a pool of IPs provided by Bright Initiative—the non-profit branch of Bright Data that supports academic and nonprofit projects with free access to data scraping tools. However, the initial version of this tool lacked robustness and extensibility: it frequently broke when ISPs updated their websites and required significant manual effort to integrate new ISPs.
To address these limitations, we built an ML-based data collection platform that queries ISP web interfaces using residential street addresses and extracts data on service availability, quality, and pricing. Bright Data’s pro-bono credits have been instrumental in this effort, enabling us to incorporate new ISPs into the framework and test the system at scale by running large volumes of address queries. BQT+ has already been applied in several policy evaluation studies, including an independent assessment of broadband availability, quality, and affordability in areas targeted by the $42.45 billion BEAD program.
Bright Data’s proxy infrastructure—especially its Web Unlocker—has been critical to scaling up data collection. Without it, we could not reliably run queries at scale on ISP websites protected by CAPTCHAs, many of which present captcha challenges that the Web Unlocker helps circumvent. Review collected by and hosted on G2.com.
