Home › Guides › The Scraping Workforce
✓ data: BLS Occupational Employment and Wage Statistics (OEWS), May 2021–May 2025; BLS Employment Projections 2025–35; O*NET 31.0; CPI-U via the BLS API, all retrieved 2 October 2026 · method · caveats · download the data
Who actually does web scraping and data collection in the US, and what are they paid? No government survey counts “web scrapers”. So we started from the job descriptions themselves. We searched O*NET, the Labor Department’s occupation database, for work that involves collecting data from the web. Then we pulled official employment and wage estimates for the ten occupations that came up, for every year from May 2021 to May 2025 and for every state. A script recomputes every number on this page from saved copies of the government files.
Read this first. These occupations do much more than scraping. A data scientist or web developer may never write a scraper, and many scrapers work under other job titles or are self-employed, which OEWS doesn’t cover. Read the figures as the size and pay of the job families that scraping work lives in, not as a headcount of people who scrape.
O*NET task statements that mention scraping or crawling the web 0 of 18,838data scientists, May 2025 262,440their median pay $120,230top-paying state for data scientists Washington $163,350
Headline numbers
- There is no “web scraper” occupation. None of the 18,838 task statements in O*NET 31.0, covering 923 occupations, mentions scraping or crawling websites. Neither do any of its 338 “emerging tasks”. Scraping shows up only in the tools lists. Selenium, the browser-automation library, is listed for 22 occupations. Microsoft Playwright, HTTrack and Screaming Frog SEO Spider are listed for 1 each.
- Data scientists are the fastest-growing group, by far. US employment rose from 105,980 in May 2021 to 262,440 in May 2025 (+147.6%). Over the same period total US employment grew +10.4%.
- But their pay has only kept pace with inflation. The median data-scientist wage rose from $100,910 to $120,230 (+19.1%). Consumer prices rose 19.4% between May 2021 and May 2025, so in real terms the change was −0.2%.
- The older web-and-code jobs are shrinking. Web developer employment fell −17.2% to 70,190, and computer programmers fell −39.6% to 92,230. Database architects grew +33.1% to 67,140.
- Database architects are the best paid of the ten occupations. Their median was $139,500 in May 2025, ahead of software developers ($135,980), data scientists ($120,230) and web developers ($92,650). All ten pay well above the all-occupation median of $50,980.
- Washington pays the most for data scientists and web developers. The median there is $163,350 for data scientists and $130,440 for web developers. California pays the most for database architects ($170,160) and has the highest median in 4 of the ten occupations. Massachusetts is in the top five for 8 of the ten.
- The gap between states is wide. Data scientists in Mississippi have a median of $69,490, less than half the Washington figure. Web developers in Arkansas have a median of $50,550.
- BLS expects data-scientist jobs to keep growing. Its 2025–35 projections have data-scientist employment growing +34.6%, with about 24,800 openings a year. Web developers are projected at +3.8% and computer programmers at −7.3%. BLS rates all ten occupations “Very high” for exposure to AI. BLS says that doesn’t mean job losses.
Employment since 2021
| Occupation | Jobs, May 2021 | Jobs, May 2025 | Change | Median, 2021 | Median, 2025 | Nominal change | Real change* |
|---|---|---|---|---|---|---|---|
| Software developers 15-1252 | 1,364,180 | 1,687,890 | +23.7% | $120,730 | $135,980 | +12.6% | −5.7% |
| Market research analysts & marketing specialists 13-1161 | 727,540 | 899,580 | +23.6% | $63,920 | $78,760 | +23.2% | +3.2% |
| Computer occupations, all other 15-1299 | 370,190 | 435,370 | +17.6% | $95,270 | $116,580 | +22.4% | +2.5% |
| Data scientists 15-2051 | 105,980 | 262,440 | +147.6% | $100,910 | $120,230 | +19.1% | −0.2% |
| Software QA analysts & testers 15-1253 | 190,120 | 186,740 | −1.8% | $98,220 | $104,300 | +6.2% | −11.1% |
| Web & digital interface designers 15-1255 | 82,380 | 113,330 | +37.6% | $79,890 | $104,000 | +30.2% | +9.0% |
| Computer programmers 15-1251 | 152,610 | 92,230 | −39.6% | $93,000 | $100,390 | +7.9% | −9.6% |
| Web developers 15-1254 | 84,820 | 70,190 | −17.2% | $77,030 | $92,650 | +20.3% | +0.7% |
| Database administrators 15-1242 | 85,870 | 69,990 | −18.5% | $96,710 | $104,620 | +8.2% | −9.4% |
| Database architects 15-1243 | 50,440 | 67,140 | +33.1% | $123,430 | $139,500 | +13.0% | −5.4% |
| All occupations | 140,886,310 | 155,495,730 | +10.4% | $45,760 | $50,980 | +11.4% | −6.7% |
* Adjusted with CPI-U (all items, US city average, not seasonally adjusted), May 2021 = 269.195, May 2025 = 321.465. Medians are annual wages for full-year, full-time work.
Year by year, the three named roles (jobs, with median wage underneath)
| Occupation | May 2021 | May 2022 | May 2023 | May 2024 | May 2025 |
|---|---|---|---|---|---|
| Data scientists | 105,980 $100,910 | 159,630 $103,500 | 192,710 $108,020 | 233,440 $112,590 | 262,440 $120,230 |
| Web developers | 84,820 $77,030 | 88,620 $78,580 | 85,350 $84,960 | 78,860 $90,930 | 70,190 $92,650 |
| Database architects | 50,440 $123,430 | 62,470 $134,870 | 59,920 $134,700 | 64,770 $135,980 | 67,140 $139,500 |
What they earn
Which states pay the most
| Occupation | Highest median wage | Lowest median wage | Most jobs | Most concentrated (jobs per 1,000) |
|---|---|---|---|---|
| Database architects | California ($170,160) Massachusetts ($161,650) Virginia ($160,360) | Mississippi ($98,420) | Texas (8,270) California (6,210) Virginia (5,260) | Virginia (1.280) |
| Software developers | California ($174,410) Washington ($166,540) New York ($166,180) | Mississippi ($95,330) | California (284,390) Texas (163,880) New York (113,510) | Washington (30.141) |
| Data scientists | Washington ($163,350) California ($141,590) Maryland ($136,370) | Mississippi ($69,490) | California (39,310) Texas (25,860) New York (23,970) | District of Columbia (3.825) |
| Computer occupations, all other | District of Columbia ($156,590) Maryland ($144,680) Colorado ($139,580) | Tennessee ($76,470) | California (80,150) Texas (57,020) Georgia (20,630) | District of Columbia (24.501) |
| Database administrators | Utah ($135,750) Massachusetts ($129,300) New Jersey ($125,860) | Maine ($70,500) | California (7,240) Texas (6,430) Virginia (5,940) | Virginia (1.445) |
| Software QA analysts & testers | California ($128,740) Washington ($128,480) Colorado ($123,690) | West Virginia ($62,300) | California (27,710) Texas (19,590) New York (10,460) | Virginia (2.474) |
| Web & digital interface designers | California ($136,110) Washington ($130,480) New York ($120,990) | Mississippi ($48,550) | California (27,090) New York (16,560) Washington (11,580) | Washington (3.261) |
| Computer programmers | Connecticut ($133,820) Massachusetts ($128,390) Washington ($125,620) | South Dakota ($58,240) | California (11,340) Texas (7,620) New York (7,020) | Alaska (1.634) |
| Web developers | Washington ($130,440) Virginia ($128,380) California ($119,570) | Arkansas ($50,550) | California (7,320) Texas (4,910) Virginia (4,590) | Washington (1.248) |
| Market research analysts & marketing specialists | Massachusetts ($101,810) Delaware ($101,270) New York ($98,580) | West Virginia ($52,340) | California (119,070) New York (80,500) Texas (63,350) | Massachusetts (10.916) |
May 2025. The 50 states and the District of Columbia; territories excluded. BLS withholds some state estimates for confidentiality or reliability. Those are left blank in the CSV and skipped in the rankings, never treated as zero. Every state and occupation is in the state CSV.
The next ten years
| Occupation | Projected job growth, 2025–35 | Openings per year | BLS AI exposure |
|---|---|---|---|
| Data scientists | +34.6% | 24,800 | Very high |
| Software developers | +10.2% | 95,300 | Very high |
| Database architects | +9.4% | 3,900 | Very high |
| Market research analysts & marketing specialists | +7.0% | 82,000 | Very high |
| Web & digital interface designers | +6.0% | 8,300 | Very high |
| Software QA analysts & testers | +5.7% | 10,700 | Very high |
| Computer occupations, all other | +5.1% | 27,000 | Very high |
| Web developers | +3.8% | 5,300 | Very high |
| Database administrators | −0.1% | 3,300 | Very high |
| Computer programmers | −7.3% | 4,400 | Very high |
Source: BLS Employment Projections 2025–35 and its AI exposure categories table. Projections cover self-employed workers too, so their base-year employment differs from the OEWS figures above. Openings include replacing workers who leave the occupation.
Method
- Choosing the occupations. We included the three roles we set out to cover: data scientists (SOC 15-2051), web developers (15-1254) and database architects (15-1243). Then we added occupations O*NET links to web-data collection:
- We searched the O*NET Web Services API for “web scraping”. It returned 118 matches, and the first six were web or online-business occupations. Their OEWS groups are Computer Occupations, All Other (which contains Web Administrators), Web & Digital Interface Designers, Market Research Analysts & Marketing Specialists (which contains Search Marketing Strategists) and Computer Programmers. We left out Online Merchants, whose OEWS group (Business Operations Specialists, All Other) is mostly unrelated work. Results after the first six were about scraping metal or food.
- We added Software Developers and Software QA Analysts & Testers, which O*NET lists with browser-automation tools (Selenium, Playwright).
- We added Database Administrators, who run the databases scraped data ends up in.
- Task search. We scanned every task statement and emerging task in the O*NET 31.0 database for words like “web scraping”, “screen scraping”, “web crawling” and “scrape” or “crawl” near “web”, “website”, “online” or “internet”. We read the matches by hand. The only “scrape” tasks are physical ones, like scraping floors or dishes.
- Employment and wages. These are BLS OEWS national and state estimates for May of each year, cross-industry, all ownerships, SOC 2018 detailed occupations. They come from the published spreadsheets. We checked the May 2025 national figures against the BLS Public Data API and they match exactly.
- Real wages. Medians are deflated with CPI-U for May of each year, fetched from the BLS API.
- Verification. A script re-reads the original BLS spreadsheets, the API responses and the O*NET files. It recomputes every number, table cell and chart point on this page and checks the CSVs.
Caveats
- Job families, not scrapers. These occupations do far more than collect web data. The figures show the size and pay of the job families where scraping work happens. They don’t count people who scrape.
- Employees only. OEWS covers wage-and-salary jobs. Freelancers, the self-employed and contractors paid through platforms aren’t included, and a lot of scraping work is done that way.
- “All other” is broad. Computer Occupations, All Other (435,370 jobs) includes web administrators but also many unrelated IT roles. Market research analysts & marketing specialists is also a large, mixed group.
- Year-to-year comparisons. OEWS estimates are survey-based and pool three years of samples. BLS cautions against reading them as a precise time series, and part of the change in a small occupation can be sampling error. Read the five-year direction, not the exact percentages.
- State pay reflects industry mix and living costs. A high state median often means more tech-industry and federal-contractor jobs, and higher prices. We haven’t adjusted for living costs.
- Some state figures are withheld. BLS suppresses some state estimates. They are missing from the rankings, not counted as zero.
- No job-posting counts. The free, openly licensed job-postings data we found (Indeed Hiring Lab, CC BY 4.0) is an index of change in posting volume. It doesn’t publish counts. We didn’t find posting counts that we could both verify and republish, so we left postings out.
- AI exposure. The AI exposure rating is BLS’s relative classification. BLS says it does not imply job loss, productivity gains or wage effects.
Download the data
- National, by occupation and year (CSV): employment and median wage, May 2021–2025, for the ten occupations and all occupations, plus the 2025–35 projections.
- By state, May 2025 (CSV, 499 rows): employment, jobs per 1,000, location quotient, median, mean and 90th-percentile wage, and each state’s pay rank.
Cite as: ProxyPicker, “The Scraping Workforce”, 2 October 2026, https://proxypicker.com/scraping-workforce/. Underlying data: U.S. Bureau of Labor Statistics (OEWS, Employment Projections, CPI). O*NET 31.0 Database by the U.S. Department of Labor, Employment and Training Administration (USDOL/ETA). Used under the CC BY 4.0 license. ProxyPicker selected the occupations and ran the task search. USDOL/ETA has not approved, endorsed or tested these modifications.
Related: Best proxies for web scraping · Web scraping law hub · Anti-Bot Census · Anti-Bot vs Proxy Patents · Proxy & Scraping Search Demand Report (what people search for, 2022–2026) · Residential IP Supply by Country (what providers claim, against internet users).
Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.