Engineering & Technology - Engineering
Software Engineer - Data Infrastructure (Pretraining Data)
IC / Practitioner
Independently auditedBuilds and operates the software the business runs on. Owns a defined area of the function and the outcomes it produces.
Audited package
Built, independently audited role package. Everything on this page is written for this seat.
Your Stack (Swappable)
This role works against whatever tools you already run. We list the tool categories the role operates in, never a named vendor, so you are never asked to replace a system that works.
- Corpus storage and manifests
- Distributed processing
- Deduplication and similarity
- Quality and safety filtering
- Tokenization and packing
- Data Governance & Catalog
- Validation and analysis
Deliverables
- Design documents
- Review checklists
- Technical standards
- Delivery and sequencing plans
Responsibilities
- Design and operate corpus-build pipelines that ingest acquired or licensed sources and produce deterministic training-ready dataset releases.
- Implement normalization, language and format handling, exact and near deduplication, quality filtering, sensitive-data controls, and source-specific exclusions.
- Build benchmark and evaluation contamination detection, quarantine, evidence review, and release gates.
- Define tokenization, document boundaries, sequence packing, sharding, mixture sampling, and read-performance contracts with training consumers.
- Preserve source, document, transform, filter, deletion, and release lineage so every included or excluded item has an explainable path.
- Evaluate corpus composition and filter effects across language, domain, source, modality, quality, safety, and licensing slices.
- Plan reprocessing and deletion propagation without creating untracked forks or non-reproducible releases.
- Partner with Acquisition, Data Platform, ML, Research, Legal, Privacy, Safety, Security, and Compute teams while retaining the prepare-data versus train-model boundary.
The essence of this seat
The Software Engineer - Data Infrastructure (Pretraining Data) is individual contributor who is the engineer who builds the corpus-processing infrastructure that turns raw web-scale and licensed sources into versioned, training-ready pretraining datasets.
Signature move
Start with a corpus release manifest that names source authority, allowed use, snapshot, transformations, filters, tokenizer, mixture, and training consumer
What good looks like
defines the corpus as a versioned compilation of sources, transforms, filters, tokenizer, mixture, and shards and proves the manifest can reproduce it
Decision rights
- - the technical design and implementation choices inside the bounded large-scale ingest, normalization, exact and near deduplication, quality and safety filtering, evaluation-set contamination controls, tokenization, sharding, mixture weighting, provenance, licensing, consent, reproducibility, and corpus-scale cost work assigned to this seat
- - the evidence, tests, safeguards, and operational artifacts needed to demonstrate this outcome: model-training teams receive reproducible, policy-compliant datasets whose content, transformations, quality tradeoffs, contamination risk, lineage, and exact release version can be explained and rebuilt
- - when a change inside the delegated scope is ready for review, canary, rollback, or further investigation
What this role does
The Software Engineer - Data Infrastructure (Pretraining Data) is individual contributor who is the engineer who builds the corpus-processing infrastructure that turns raw web-scale and licensed sources into versioned, training-ready pretraining datasets. The role is accountable for the engineering work required so model-training teams receive reproducible, policy-compliant datasets whose content, transformations, quality tradeoffs, contamination risk, lineage, and exact release version can be explained and rebuilt. Its working domain includes large-scale ingest, normalization, exact and near deduplication, quality and safety filtering, evaluation-set contamination controls, tokenization, sharding, mixture weighting, provenance, licensing, consent, reproducibility, and corpus-scale cost. A strong practitioner turns that domain into explicit contracts, measurable evidence, controlled changes, and clean hand-offs. The role reasons from the actual artifact and failure boundary rather than from a fashionable tool. It makes assumptions visible, tests the load-bearing ones, and distinguishes what is Verified, Inferred, and Not-yet-verified. The role remains at IC altitude: it owns technical execution and bounded decisions, advises leaders and neighboring specialists, and escalates product, people, budget, policy, legal, risk, and cross-organizational calls. The nearest boundaries are deliberately strict. Distinct from Software Engineer - Data Platform: this seat owns the specialized pretraining corpus build and release path; that seat owns shared ingestion, processing, catalog, and storage primitives used across many data products. Distinct from Software Engineer - Data Acquisition / Web Crawling: this seat normalizes, deduplicates, filters, tokenizes, mixes, and versions acquired material; that seat discovers and gathers the raw source under crawl and acquisition policy. Distinct from Machine Learning Engineer: this seat prepares and releases the training corpus with quality and lineage evidence; that seat defines and runs model training, optimization, and model evaluation. Those titles still receive their own complete packages under the 1:1 policy.
Best-fit customer
engineering teams that want consistent judgement applied to design, review, and delivery
Need more than one role?
Buying the whole team or the complete department lowers your effective cost per role at every tier of the ladder.