Senior Data Engineer

Experience level: mid senior level.

Data Pipeline Development

• Design, build, and maintain  ETL/ELT pipelines  in  Databricks  to ingest, clean, and transform data from diverse product sources.

• Construct  gold layer tables  in the  Lakehouse  architecture that serve both machine learning model training and real-time APIs.

• Monitor data quality, lineage, and reliability using Databricks best practices.

AI-Driven Data Access Enablement

• Collaborate with AI/ML teams to ensure data is modeled and structured to support  natural language prompts  and  semantic retrieval  using 1st and 3rd party data sources, vector search and Unity Catalog metadata.

• Help build data interfaces and agent tools to interact with structured data and AI agents to retrieve and analyze customer data with role-based permissions.

API & Serverless Backend Integration

• Work with backend engineers to design and implement  serverless APIs  (e.g., via AWS Lambda with TypeScript) that expose gold tables to frontend applications.

• Ensure APIs are performant, scalable, and designed with data security and compliance in mind.

• Utilize Databricks and other APIs to implement provisioning, deployment, security and monitoring frameworks for scaling up data pipelines, AI endpoints, and security models for multi-tenancy.

• 3+ years of experience as a  Data Engineer  or related role in an agile, distributed team environment with a quantifiable impact on business or technology outcomes.

• Proven expertise with  Databricks , including job and workflow orchestration, change data capture and medallion architecture.

• Proficiency in  Spark  or Scala for data wrangling and transformation on a wide variety of data sources and structures.

• Practitioner of  CI/CD best practices , test-driven development and familiarity with the MLOps / AIOps lifecycles.

• Proven ability to work in an agile environment with product managers, front-end engineers, and data scientists.

Preferred Skills

• Familiarity with  AWS Lambda  (Node.js/TypeScript preferred) and  API Gateway  or equivalent serverless platforms, knowledge of  API design  principles and working with RESTful or GraphQL endpoints.

• Exposure to  React-based frontend architecture  and the implications of backend data delivery on UI/UX performance – including end-to-end telemetry to measure performance and accuracy for the end-user experience.

• Experience with A/B testing, experiment and inference logging and analytics.

We are a multinational technology consulting firm. We help companies and corporations scale their operations, achieve technology innovation, elevate their brand and transform their business model.

We are here to challenge the status quo, flip the script, and blur all the lines in order to create customized end-to-end tech solutions, from software to hardware. We are a team of over 500 engineers from around the world with one shared goal: to leverage and crisscross technology, creative thinking, and industry-specific expertise to help our customers become and remain high performers in their industries. Basically, we take care of it all from A to Z.

Our expert engineers have contributed to 8 US patents and developed award-winning innovative tech solutions, serving 80M+ users for over 100 clients worldwide, including top US Fortune 500 companies.