M
Microsoftvia muse
AI Data Engineer (contract)
Redmond, WA, US$75 - $85/hrPosted 6d ago
Data EngineerMid LevelContract
About the Role
**AI Data Engineer (Contract)**
*Please note that this is a contract role providing services to Microsoft through external staffing partners of Allegis Global Solutions. If you are selected for this role, you will be employed by AGS and will not be an employee of Microsoft.*
**Summary:**
We are seeking a hands\-on Data Pipeline Engineer to build and operate Python\-based workflows that transform large document and text datasets into high\-quality data for AI model training. This role sits within an applied science team that bridges research, innovation, and production to develop and train cutting\-edge models. The ideal candidate brings strong Python programming skills, proven experience processing datasets that exceed single\-machine memory, and practical expertise managing data and processing workflows in Azure. Critically, candidates must have hands\-on experience building datasets from the ground up — not simply working with pre\-existing ones.
**Job Responsibilities:**
* Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication
* Run and monitor large\-scale batch processing jobs, investigate failures, and implement reliable retry and recovery mechanisms
* Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies
* Prepare training\-ready datasets through filtering, text segmentation, tokenization, and conversion into agreed data formats
* Manage bulk uploads, downloads, and synchronization between local processing environments and Azure storage, verifying transfer completeness and data integrity
* Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability
* Optimize processing throughput, memory usage, storage consumption, and cloud costs using batch, parallel, or distributed processing approaches
* Track dataset statistics, processing coverage, quality metrics, and exceptions, and produce clear operational reports
* Collaborate with applied scientists and model\-training teams to translate data requirements into reproducible processing workflows
* Maintain tested, documented, version\-controlled code and follow established data access, security, and governance requirements
**Requirements:**
* Minimum of five years of hands\-on, data\-related work experience
* Demonstrated experience in data engineering, data pipeline development, or a data science role with substantial production programming responsibilities
* Proven track record of creating original datasets, not solely consuming or transforming pre\-existing ones
* Background in data science, mathematics, computer science, or a closely related field
* Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors
* Hands\-on experience building datasets from the ground up, including original dataset creation and curation
* Experience processing large structured and unstructured datasets using memory\-efficient, parallel, or distributed techniques
* Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting
* Proficiency with Python data\-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet
* Working knowledge of SQL, Git, Linux environments, and command\-line tools
* Strong troubleshooting skills, attention to data quality, and ability to communicate technical issues clearly
* Experience with OCR pipelines, scanned documents, multilingual text, or document\-layout processing (preferred)
* Experience with Azure Databricks, Azure Machine Learning, Azure Data Factory, or workflow orchestration tools (preferred)
* Familiarity with Hugging Face Datasets, tokenization, near\-duplicate detection, and training/evaluation d
*Please note that this is a contract role providing services to Microsoft through external staffing partners of Allegis Global Solutions. If you are selected for this role, you will be employed by AGS and will not be an employee of Microsoft.*
**Summary:**
We are seeking a hands\-on Data Pipeline Engineer to build and operate Python\-based workflows that transform large document and text datasets into high\-quality data for AI model training. This role sits within an applied science team that bridges research, innovation, and production to develop and train cutting\-edge models. The ideal candidate brings strong Python programming skills, proven experience processing datasets that exceed single\-machine memory, and practical expertise managing data and processing workflows in Azure. Critically, candidates must have hands\-on experience building datasets from the ground up — not simply working with pre\-existing ones.
**Job Responsibilities:**
* Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication
* Run and monitor large\-scale batch processing jobs, investigate failures, and implement reliable retry and recovery mechanisms
* Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies
* Prepare training\-ready datasets through filtering, text segmentation, tokenization, and conversion into agreed data formats
* Manage bulk uploads, downloads, and synchronization between local processing environments and Azure storage, verifying transfer completeness and data integrity
* Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability
* Optimize processing throughput, memory usage, storage consumption, and cloud costs using batch, parallel, or distributed processing approaches
* Track dataset statistics, processing coverage, quality metrics, and exceptions, and produce clear operational reports
* Collaborate with applied scientists and model\-training teams to translate data requirements into reproducible processing workflows
* Maintain tested, documented, version\-controlled code and follow established data access, security, and governance requirements
**Requirements:**
* Minimum of five years of hands\-on, data\-related work experience
* Demonstrated experience in data engineering, data pipeline development, or a data science role with substantial production programming responsibilities
* Proven track record of creating original datasets, not solely consuming or transforming pre\-existing ones
* Background in data science, mathematics, computer science, or a closely related field
* Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors
* Hands\-on experience building datasets from the ground up, including original dataset creation and curation
* Experience processing large structured and unstructured datasets using memory\-efficient, parallel, or distributed techniques
* Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting
* Proficiency with Python data\-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet
* Working knowledge of SQL, Git, Linux environments, and command\-line tools
* Strong troubleshooting skills, attention to data quality, and ability to communicate technical issues clearly
* Experience with OCR pipelines, scanned documents, multilingual text, or document\-layout processing (preferred)
* Experience with Azure Databricks, Azure Machine Learning, Azure Data Factory, or workflow orchestration tools (preferred)
* Familiarity with Hugging Face Datasets, tokenization, near\-duplicate detection, and training/evaluation d