Microsoft
Senior Data Engineer
Microsoft, Little Ferry, New Jersey, us, 07643
Microsoft Cloud Operations & Innovation (CO+I)
is the engine that powers Microsoft cloud services through the operation of our unified global Datacenters enabling ~30% of Microsoft revenue through Commercial Cloud ($38 billion in FY20 Q1). The
Cloud Infrastructure Health Team
in
CO+I
is focused on improving Customer Availability, Data center Safety, Capacity and helping optimize the utilization of Datacenter resources using telemetry and Insights. Our systems analyze petabyte scale telemetry data from Datacenter critical environments and secondary signals in near real time and offline that enable time-sensitive insights directly impacting Cloud Operations.
We are looking for a
Senior Data Engineer
to work closely with our software engineers, product managers, data scientists, and analysts to build data pipelines and data products, real-time detections using Machine Learning and Heuristics. As a
Senior Data Engineer , you'll be at the forefront of Microsoft’s cloud computing transformation. You will be agile, working with state of the art distributed systems that deal with near real time detection on petabyte scale telemetry using Machine Learning and traditional software to help deliver on Cloud Availability and Safety Goals. You'll be helping to build Petabyte scale Big Data Pipelines to ingest and process Data center Critical Environment Telemetry and draw deep insights designed to improve customer availability of Microsoft Cloud, safety and Infrastructure optimization.
In alignment with our Microsoft values, we are committed to cultivating an inclusive work environment for all employees to positively impact our culture every day.
ResponsibilitiesShip high-quality, well-tested, secure, and maintainable code.Design, develop, and maintain data pipelines and back-end services for real-time decisioning, reporting, optimization, data collection, and related functions.Manage automated unit and integration test suites.Work collaboratively and communicate effectively with a small, motivated team of engineers and product managers.Experiment with and recommend new technologies that simplify or improve the tech stack.Participate in an on-call rotation and work occasional off-hours.QualificationsRequired/Minimum Qualifications:Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 4+ years experience in Apache Spark, Structured Streaming, Azure, software development, data modeling or data engineering work.OR Master's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 3+ years experience in business analytics, data science, software development, data modeling or data engineering work.OR equivalent experience.Other Requirements:Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to, the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.Preferred/Additional Qualifications:Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 8+ years experience in Apache Spark, Structured Streaming, Azure, software development, data modeling or data engineering work.OR Master's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 6+ years experience in business analytics, data science, software development, data modeling or data engineering work.OR equivalent experience.Architecting, building, and maintaining end-to-end, high-throughput data systems and their supporting services.Designing efficient data structures and database schemas and working with distributed systems architecture.Incorporating data processing and workflow management tools into pipeline design.Utilizing a variety of data stores, including data warehouses, relational database systems, in-memory caches, and searchable document databases.Experience working with large data sets using SQL/Azure Data Lake/Spark, etc., to derive actionable insights is highly desirable.Developing for continuous integration and automated deployments.Using profiling tools, debugging logs, performance metrics, and other data sources to make code- and application-level improvements.Data Engineering IC4 - The typical base pay range for this role across the U.S. is USD $112,000 - $218,400 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $145,800 - $238,600 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
Microsoft is an equal opportunity employer. Consistent with applicable law, all qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.#J-18808-Ljbffr
is the engine that powers Microsoft cloud services through the operation of our unified global Datacenters enabling ~30% of Microsoft revenue through Commercial Cloud ($38 billion in FY20 Q1). The
Cloud Infrastructure Health Team
in
CO+I
is focused on improving Customer Availability, Data center Safety, Capacity and helping optimize the utilization of Datacenter resources using telemetry and Insights. Our systems analyze petabyte scale telemetry data from Datacenter critical environments and secondary signals in near real time and offline that enable time-sensitive insights directly impacting Cloud Operations.
We are looking for a
Senior Data Engineer
to work closely with our software engineers, product managers, data scientists, and analysts to build data pipelines and data products, real-time detections using Machine Learning and Heuristics. As a
Senior Data Engineer , you'll be at the forefront of Microsoft’s cloud computing transformation. You will be agile, working with state of the art distributed systems that deal with near real time detection on petabyte scale telemetry using Machine Learning and traditional software to help deliver on Cloud Availability and Safety Goals. You'll be helping to build Petabyte scale Big Data Pipelines to ingest and process Data center Critical Environment Telemetry and draw deep insights designed to improve customer availability of Microsoft Cloud, safety and Infrastructure optimization.
In alignment with our Microsoft values, we are committed to cultivating an inclusive work environment for all employees to positively impact our culture every day.
ResponsibilitiesShip high-quality, well-tested, secure, and maintainable code.Design, develop, and maintain data pipelines and back-end services for real-time decisioning, reporting, optimization, data collection, and related functions.Manage automated unit and integration test suites.Work collaboratively and communicate effectively with a small, motivated team of engineers and product managers.Experiment with and recommend new technologies that simplify or improve the tech stack.Participate in an on-call rotation and work occasional off-hours.QualificationsRequired/Minimum Qualifications:Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 4+ years experience in Apache Spark, Structured Streaming, Azure, software development, data modeling or data engineering work.OR Master's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 3+ years experience in business analytics, data science, software development, data modeling or data engineering work.OR equivalent experience.Other Requirements:Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to, the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.Preferred/Additional Qualifications:Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 8+ years experience in Apache Spark, Structured Streaming, Azure, software development, data modeling or data engineering work.OR Master's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 6+ years experience in business analytics, data science, software development, data modeling or data engineering work.OR equivalent experience.Architecting, building, and maintaining end-to-end, high-throughput data systems and their supporting services.Designing efficient data structures and database schemas and working with distributed systems architecture.Incorporating data processing and workflow management tools into pipeline design.Utilizing a variety of data stores, including data warehouses, relational database systems, in-memory caches, and searchable document databases.Experience working with large data sets using SQL/Azure Data Lake/Spark, etc., to derive actionable insights is highly desirable.Developing for continuous integration and automated deployments.Using profiling tools, debugging logs, performance metrics, and other data sources to make code- and application-level improvements.Data Engineering IC4 - The typical base pay range for this role across the U.S. is USD $112,000 - $218,400 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $145,800 - $238,600 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
Microsoft is an equal opportunity employer. Consistent with applicable law, all qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.#J-18808-Ljbffr