Your Role:

We are seeking a highly skilled and motivated Senior Data Engineer to join our growing team at Nayya. In this role, you will help with the design and implementation of scalable data systems and pipelines that power our Claims product and central data strategy. You will work on building batch, event processing, and stream processing infrastructure, enhancing our data enrichment services, developing a robust, de-identified analytics platform for our Data Science, BI, and Analytics teams to consume, and enabling our entire organization to make data driven decisions. We are looking for an expert who thrives in an environment that values impatience, excellence, resilience, and couragea leader ready to make an immediate impact on our data infrastructure in a fast-paced, high-growth environment.

As a Senior Data Engineer, you will play a key role in shaping our data systems' and claims product architecture, reliability, and performance while fostering innovation and collaboration across teams. This position provides an exciting opportunity to drive technical strategy and lead efforts to solidify and scale our data infrastructure.

Key Responsibilities:

Technical Leadership & Data Infrastructure

• Claim Product Development: Help implement new partners on our claims product and develop standardized, low-tech onboarding solutions for new clients.
• Centralized Data Strategy: Develop a single source of truth for organizational data, driving data validation, governance, and improved access for analytical and operational use.
• Build, Improve, and Maintain Data Systems: Help develop scalable data pipelines that handle high-volume batch and streaming data.
• Data API and Eventing Development: Enhance and maintain APIs and event driven architecture to provide efficient and reliable access to internal and external data consumers.
• Data Enrichment & Integration: Implement data enrichment solutions at scale that interface with third-party data sources to enhance product capabilities.
• Analytics & Reporting Platform: Improve our reporting and analytics platform while treating security and compliance as a top priority.

Collaboration & Mentorship

• Cross-Functional Collaboration: Work closely with product, engineering, business, and infrastructure teams to design solutions that meet evolving business and technical needs. Advocate for data-driven decision making.
• Mentor and Develop: Provide guidance and mentorship to engineers, fostering a culture of continuous learning and growth.
• Lead with documentation: Identify and evaluate our current processes, documentation, workflows and governance and make recommendations and plans for improvements.

Continuous Improvement

• Optimize Performance: Focus on tuning, performance testing, and optimization of the data platform.
• Innovate with Agility: Embrace a growth mindset, iterating on data infrastructure and processes to ensure scalability and reliability.
• Ensure Security and Scalability: Identify gaps and risk in current infrastructure to solidify the data platform.

Qualifications

• 4+ years of experience in data engineering, data infrastructure, or related roles.
• Strong experience with Python and PySpark.
• Strong experience with RDBMS.
• Proficiency with workflow orchestration tools (Airflow, Dagster, etc.).
• Experience implementing data pipelines using Apache Spark, AWS Glue, or EMR.
• Expertise in SQL optimization, query performance tuning, and data warehousing.
• Experience with AWS suite of data engineering managed services and OSS tools.
• Experience with monitoring and observability frameworks and tools.
• Familiarity with data quality measures, tools, and frameworks.
• Ability to identify tradeoffs for warehousing vs data lake infrastructure and applying solutions to the appropriate use case.
• Ability to communicate highly technical topics to non-technical stakeholders.
• Familiar with common pitfalls in high volume, partitioned data ingestion pipelines such as orphaned records and table locks.

Preferred Qualifications:

• Experience with Apache Hudi or similar data lake platforms.
• Experience with infrastructure as code tools such as Terraform.
• Experience with Redshift.
• Experience with federated query engines.
• Experience with data catalogues.
• Experience with claims data.
• Experience with MLOps engineering and best practices.
• Experience with data governance over PHI and other sensitive information.
• Experience in fast-paced startup environments or high-growth companies.

The salary range for New York based candidates for this role is $135,000-$175,000. We use a location factor to adjust this range for candidates that are located outside of geographic region of our New York office. Placement within the salary band is determined based on experience. 

#LI-JS1

#LI-HYBRID