Sr. Data Engineer

Title: Sr. Data EngineerLocation: Remote- ESTDuration: 22 Month ContractCompensation: $90-$95/hrWork Requirements: US Citizen, GC Holders or Authorized to Work in the USWe are seeking an experienced AI/LLM Data Engineer to build and maintain the data pipeline for our Generative AI platform. The ideal candidate will be well-versed in the latest Large Language Model (LLM) technologies and have a strong background in data engineering, with a focus on Retrieval-Augmented Generation (RAG) and knowledge-base techniques. This role sits in the AI COE. As a AI/LLM Data Engineer (you will report into the Director, AI Solutions & Development who oversees the AI COE.You will work on highly visible strategic projects, collaborating with cross-functional teams to define requirements and deliver high-quality AI solutions. The ideal candidate will have a passion for Generative AI and LLMs, with a proven track record of delivering innovative AI applications.Responsibilities: Design, implement, and maintain an end-to-end multi-stage data pipeline for LLMs, including Supervised Fine Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) data processesIdentify, evaluate, and integrate diverse data sources and domains to support the Generative AI platformDevelop and optimize data processing workflows for chunking, indexing, ingestion, and vectorization for both text and non-text dataBenchmark and implement various vector stores, embedding techniques, and retrieval methodsCreate a flexible pipeline supporting multiple embedding algorithms, vector stores, and search types (e.g., vector search, hybrid search)Implement and maintain auto-tagging systems and data preparation processes for LLMsDevelop tools for text and image data crawling, cleaning, and refinementCollaborate with cross-functional teams to ensure data quality and relevance for AI/ML modelsWork with data lake house architectures to optimize data storage and processingIntegrate and optimize workflows using Snowflake and various vector store technologiesSkillset / Experience: Master's deg in Computer Science, Data Science, etc3-5 years of work experience in data engineering, preferably in AI/ML contextsProficiency in Python, JSON, HTTP, and related toolsStrong understanding of LLM architectures, training processes, and data requirementsExperience with RAG systems, knowledge base construction, and vector databasesFamiliarity with embedding techniques, similarity search algorithms, and information retrieval conceptsHands-on experience with data cleaning, tagging, and annotation processes (both manual and automated)Knowledge of data crawling techniques and associated ethical considerationsStrong problem-solving skills and ability to work in a fast-paced, innovative environmentFamiliarity with Snowflake and its integration in AI/ML pipelinesExperience with various vector store technologies and their applications in AIUnderstanding of data lakehouse concepts and architecturesExcellent communication, collaboration, and problem-solving skills.Ability to translate business needs into technical solutions.Passion for innovation and a commitment to ethical AI development.Experience building LLMs pipeline using framework like LangChain, LlamaIndex, Semantic Kernel, OpenAI functions.Familiar with different LLM parameters like temperate, top-k, and repeat penalty, and different LLM outcome evaluation data science metrics and methodologies.Nice to Have: Experience with popular LLM/ RAG frameworksFamiliarity with distributed computing platforms (e.g., Apache Spark, Dask)Knowledge of data versioning and experiment tracking toolsExperience with cloud platforms (AWS, GCP, or Azure) for large-scale data processingUnderstanding of data privacy and security best practicesPractical experience implementing data lakehouse solutionsProficiency in optimizing queries and data processes in Snowflake or DatabricksHands-on experience with different vector store technologiesOur benefits package includes: Comprehensive medical benefitsCompetitive pay401(k) Retirement plan...and much more!About INSPYR Solutions:Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com.INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics. In addition to federal law requirements, INSPYR Solutions complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities.