Magic, Inc
Software Engineer - Pretraining Data
Magic, Inc, San Francisco, California, United States, 94199
Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pre-training, domain-specific RL, ultra-long context, and test-time compute to achieve this goal.About the role:
As a Software Engineer working on our pretraining data, you write efficient and robust pipelines for giant, multimodal datasets. You will develop and optimize web scraping techniques to harvest and maintain data at internet-scale.What you might work on:
Design & implement multimodal (video, audio, text etc) web crawlers for scraping and indexing petabytes of dataCreate large scale data processing pipelines using tools like Ray, Apache Spark, Apache Flink, Google BigQuery etc.Implement and scale deduplication techniques across modalities and apply heuristic and model-based techniques for parsing and filtering crawled dataIdentify new data sources for inclusion in pre/post-training datasetsWhat we’re looking for:
Strong proficiency in distributed computing and parallel processing techniquesObsession with details, reliability, and good testing to ensure data quality and integrityExperience with designing and maintaining high-performance, scalable data architecturesAbility to design, develop and operate an LLM data pipeline from web scraping to data loadingOur culture:
Integrity. Words and actions should be alignedHands-on. At Magic, everyone is buildingTeamwork. We move as one team, not N individualsFocus. Safely deploy AGI. Everything else is noiseQuality. Magic should feel like magicCompensation, benefits and perks (US):
Annual salary range: $100K - $900KEquity is a significant part of total compensation, in addition to salary401(k) plan with 6% salary matchingGenerous health, dental and vision insurance for you and your dependentsUnlimited paid time offOption to work in-person in SF or remotelyVisa sponsorship and relocation stipend to bring you to SFA small, fast-paced, highly focused team
#J-18808-Ljbffr
As a Software Engineer working on our pretraining data, you write efficient and robust pipelines for giant, multimodal datasets. You will develop and optimize web scraping techniques to harvest and maintain data at internet-scale.What you might work on:
Design & implement multimodal (video, audio, text etc) web crawlers for scraping and indexing petabytes of dataCreate large scale data processing pipelines using tools like Ray, Apache Spark, Apache Flink, Google BigQuery etc.Implement and scale deduplication techniques across modalities and apply heuristic and model-based techniques for parsing and filtering crawled dataIdentify new data sources for inclusion in pre/post-training datasetsWhat we’re looking for:
Strong proficiency in distributed computing and parallel processing techniquesObsession with details, reliability, and good testing to ensure data quality and integrityExperience with designing and maintaining high-performance, scalable data architecturesAbility to design, develop and operate an LLM data pipeline from web scraping to data loadingOur culture:
Integrity. Words and actions should be alignedHands-on. At Magic, everyone is buildingTeamwork. We move as one team, not N individualsFocus. Safely deploy AGI. Everything else is noiseQuality. Magic should feel like magicCompensation, benefits and perks (US):
Annual salary range: $100K - $900KEquity is a significant part of total compensation, in addition to salary401(k) plan with 6% salary matchingGenerous health, dental and vision insurance for you and your dependentsUnlimited paid time offOption to work in-person in SF or remotelyVisa sponsorship and relocation stipend to bring you to SFA small, fast-paced, highly focused team
#J-18808-Ljbffr