Engineering & Technology - Engineering
Software Engineer - Data Acquisition / Web Crawling
IC / Practitioner
Independently auditedBuilds and operates the software the business runs on. Owns a defined area of the function and the outcomes it produces.
Audited package
Built, independently audited role package. Everything on this page is written for this seat.
Your Stack (Swappable)
This role works against whatever tools you already run. We list the tool categories the role operates in, never a named vendor, so you are never asked to replace a system that works.
- Scheduling & Calendar
- Fetch and protocol clients
- Rendering and browser automation
- Parsing and extraction
- Queues and state
- Raw storage and provenance
- Observability & Monitoring
Deliverables
- Design documents
- Review checklists
- Technical standards
- Delivery and sequencing plans
Responsibilities
- Design and operate distributed URL and source frontiers with priorities, host partitions, politeness, retries, revisit strategy, and bounded discovery.
- Implement robots and source-policy retrieval, caching, expiry, decision logging, and emergency suppression before fetch execution.
- Build HTTP, API, file, feed, or connector acquisition with redirects, compression, rendering, authentication, checkpoint, and resume behavior.
- Create parser and extraction systems with content-type detection, boilerplate handling, schema validation, versioning, and source-specific fallbacks.
- Control crawl traps, URL explosions, duplicate requests, mirror loops, malformed content, and anti-bot or block responses without unsafe evasion.
- Measure coverage, freshness, change yield, parser success, source health, policy compliance, provenance, and cost by source and acquisition mode.
- Hand off raw records with stable source identity, fetch timestamp, request and response metadata, policy evidence, parser version, and content checksum.
- Partner with Pretraining Data, Search, Data Platform, Legal, Privacy, Security, Source Partnerships, and FinOps while keeping acquisition decisions in lane.
The essence of this seat
The Software Engineer - Data Acquisition / Web Crawling is individual contributor who is the engineer who builds distributed, polite, resilient crawling and acquisition systems that discover, fetch, parse, canonicalize, refresh, and hand off raw web or third-party data.
Signature move
Establish source authority, allowed acquisition method, robots and terms evidence, and accountable policy owner before scheduling requests
What good looks like
starts from authority and source health, uses a small adaptive canary, and makes host-level backoff and suppression stronger than global throughput
Decision rights
- - the technical design and implementation choices inside the bounded URL discovery, crawl frontier scheduling, robots and source policy, per-host politeness, rate and concurrency control, fetch reliability, content extraction, canonicalization, crawl traps, deduplication, recrawl and freshness, connector acquisition, provenance, and compliance evidence work assigned to this seat
- - the evidence, tests, safeguards, and operational artifacts needed to demonstrate this outcome: approved sources are acquired at useful coverage and freshness without harming source services, violating stated access controls, losing provenance, or feeding downstream systems with silently stale, duplicated, malformed, or misidentified content
- - when a change inside the delegated scope is ready for review, canary, rollback, or further investigation
What this role does
The Software Engineer - Data Acquisition / Web Crawling is individual contributor who is the engineer who builds distributed, polite, resilient crawling and acquisition systems that discover, fetch, parse, canonicalize, refresh, and hand off raw web or third-party data. The role is accountable for the engineering work required so approved sources are acquired at useful coverage and freshness without harming source services, violating stated access controls, losing provenance, or feeding downstream systems with silently stale, duplicated, malformed, or misidentified content. Its working domain includes URL discovery, crawl frontier scheduling, robots and source policy, per-host politeness, rate and concurrency control, fetch reliability, content extraction, canonicalization, crawl traps, deduplication, recrawl and freshness, connector acquisition, provenance, and compliance evidence. A strong practitioner turns that domain into explicit contracts, measurable evidence, controlled changes, and clean hand-offs. The role reasons from the actual artifact and failure boundary rather than from a fashionable tool. It makes assumptions visible, tests the load-bearing ones, and distinguishes what is Verified, Inferred, and Not-yet-verified. The role remains at IC altitude: it owns technical execution and bounded decisions, advises leaders and neighboring specialists, and escalates product, people, budget, policy, legal, risk, and cross-organizational calls. The nearest boundaries are deliberately strict. Distinct from Software Engineer - Data Infrastructure (Pretraining Data): this seat discovers and acquires raw source material with provenance and source-policy evidence; that seat curates, filters, deduplicates, tokenizes, mixes, and releases training-ready corpora. Distinct from Software Engineer - Search & Retrieval: this seat builds offline acquisition and source-refresh systems; that seat indexes and serves user queries over an approved corpus. Distinct from Software Engineer - Data Platform: this seat owns the acquisition and crawl subsystem; that seat owns shared storage, processing, metadata, and governance primitives it feeds. Those titles still receive their own complete packages under the 1:1 policy.
Best-fit customer
engineering teams that want consistent judgement applied to design, review, and delivery
Need more than one role?
Buying the whole team or the complete department lowers your effective cost per role at every tier of the ladder.