Principal Engineer, Data Platform
- Salary
- $275,000 - $350,000
- Level
- Principal
- Work type
- On-site · New York
Not stated in the posting: visa.
Open. First seen 18 November 2026.
About the role
Traversal's standard text
About Traversal
Traversal is the AI Site Reliability Engineer (AI SRE) for the enterprise.
Production complexity was already outpacing what engineering teams could manage manually, and AI-generated code is accelerating that gap. Traversal is built for that challenge, autonomously understanding and reasoning across even the largest, most complex production environments to diagnose, fix, and prevent incidents. Our mission is to free engineers from endless firefighting and give them more time to focus on creative, high-impact work.
Today, Traversal operates in mission-critical environments at some of the world’s largest enterprises, including PepsiCo, Nubank, Harvey, and DigitalOcean. Our roots remain deeply embedded in AI research, and we’ve brought together researchers from institutions including MIT, Harvard, Berkeley, Columbia, and Cornell with world-class technical staff and operators from companies like Google, Meta, Datadog, ServiceNow, and Citadel Securities to take on one of the hardest problems for AI to solve. Traversal is backed by Sequoia Capital, Kleiner Perkins, Redpoint Ventures, and Khosla Ventures.
The Role
The Traversal AI SRE agent's performance and accuracy are hugely dependent on the data it reasons over. When Traversal investigates a production incident at a Fortune 100 company, we rely on having the right data, in the right shape, in the right place, and at the right time, to deliver the best and fastest outcomes to our customers. This is the crucially important role of Traversal's Production World Model™ and petabyte-scale data platform: hundreds of billions of log lines a day from a single customer, ingested at hundreds of megabytes per second, and queryable by the agent within minutes of being emitted.
We're hiring a Principal Engineer, Data Platform, to own the technical direction of that platform. You'll define how data enters Traversal, how it's stored and queried, and how it's transformed into the derived products our agents and researchers rely on — across SaaS and BYOC deployments in some of the most demanding enterprise environments in the world.
This is a hands-on technical leadership role: you'll shape roadmaps and write the design docs that set the direction, write the code that proves it out, and raise the bar for the engineers who build alongside you.
This is also a rare seat where data infrastructure meets frontier AI. The agent is the platform's most demanding customer, and AI Research is its closest partner. You'll see firsthand how an AI agent reasons over production data, and your decisions will directly shape how well it does. At our stage, a Principal Engineer doesn't work from a ticket queue. You'll work directly with leadership on what we build and why, sit in front of our largest customers, and learn how an enterprise AI company is built from the inside.
What You'll Do
- Own the data platform architecture. Define the long-term architecture across our four data paths — direct push ingestion, scheduled pulls from third-party observability platforms, non-telemetry data, and transformation pipelines — including the interfaces, invariants, and evolution path that let them scale together.
- Set the storage and query strategy. Decide what our storage layer looks like at 10–100x today's scale: time-series, columnar/OLAP, and lakehouse tradeoffs; retention and cost; performance commitments for the agent's key queries.
- Serve the agent as a first-class workload. An AI agent is the platform's most demanding consumer. Design the query interfaces it depends on: performance commitments for its key tool queries, freshness guarantees for the data it reasons over, and safe sandboxing for the ad-hoc queries an LLM will inevitably write.
- Build the data foundation for AI research. Work side by side with our AI Research team to give the agent and its auto-research loops the data they need: investigation history, evaluation datasets, and the derived signals that make the agent measurably better. You'll learn how a production AI agent is built, evaluated and improved, and you'll see your data decisions show up in its accuracy.
- Shape the company, not just the platform. Work directly with engineering and company leadership on the technical roadmap and how it serves the business. Join architecture reviews and deployments with Fortune 100 customers. Bring what you learn from Product, Forward Deployed Engineering and Go-to-Market back into what we build.
- Build the orchestration and transformation backbone. Lead the framework for scheduled, idempotent, observable jobs that ingest non-telemetry context and produce derived data — materialized views, rollups, entity graphs, and LLM-based summarization and enrichment that's cached ahead of investigations rather than computed during them.
- Own reliability, performance, and cost end to end. Define and meet SLOs for availability, data freshness, and query latency; build systems that ship with their monitoring, runbooks, and safe rollouts; and lead the technical response when things break.
- Govern customer data. Design redaction, tenant isolation, and data-residency controls that satisfy the security and compliance teams of the world's largest banks, payment companies, and tech companies.
- Multiply the team. Mentor senior engineers, and set standards and reference implementations that teams beyond Data Platform adopt.
What We're Looking For
- Senior-level depth in building and operating data infrastructure in production (typically 10+ years), including a track record of setting architecture for a data platform used by multiple teams.
- Deep distributed-systems expertise in streaming data, analytical storage, and query engines: Kafka (or equivalent), ClickHouse/Doris/Druid/Pinot, data lakehouse technologies, and building scalable and reliable multi-tenant data systems.
- Strong systems programming in Rust (our primary data platform language) or comparable experience in Go, plus comfort in Python.
- Experience shipping on Kubernetes, Helm, Terraform, and AWS, ideally into environments you don't fully control (BYOC, on-prem, or regulated customer clouds).
- A record of leading multi-quarter migrations and platform transformations through to adoption and deprecation, not just design.
- You still write production code regularly, and can walk us through recent work you personally shipped.
- Sound cost judgment: you reason about tradeoffs between performance, cost, durability, and simplicity with real numbers, and you know what your systems cost to run.
- Clear technical writing and judgment: you can make a crisp tradeoff call, get alignment without authority, and hold your own in an architecture review with an enterprise customer.
Nice to Have
- Experience building the backend of an observability, logging, APM, or SIEM product.
- Fluency with OpenTelemetry/OTLP and the data models and APIs of major observability vendors (Datadog, Splunk, Elastic, Prometheus, Dynatrace).
- Experience with workflow orchestration (Airflow, Dagster, Temporal) and data pipeline frameworks.
- Experience building LLM-powered data pipelines: enrichment, summarization, embeddings, and controlling their cost.
- Background in data governance: PII redaction, data residency, and regulated industries such as financial services.
- Contributions to open-source data infrastructure.
Compensation
We offer competitive compensation, startup equity, health insurance, and additional benefits. The U.S. base salary range for this full-time, in-person role in New York is $275,000 to $350,000 USD, plus equity and benefits. Our salary ranges are based on location, level, and role. Individual compensation is determined by experience, skills, and job-related knowledge.
Why You Should Join Us
If you've mastered data infrastructure and want to be at the center of AI, rather than next to it, this is the role.
Traversal's standard text
Traversal is a place to take on hard, meaningful problems with real ownership from day one. You’ll work alongside people who challenge you to grow, learn constantly, and help define a new category of infrastructure software. We think long term, move quickly, and hold a high bar without taking ourselves too seriously.
We offer competitive salary and equity packages, health insurance, fertility benefits, a great tech setup stipend and flexible time off. Plus in-office snacks, team happy hours and outings, an annual company offsite, and plenty of built in time to collaborate across teams.
Traversal is fully in-office, 5 days a week, based in New York near Madison Square Park. We have a collaborative, hard-working culture and are energized by building the future of AI-powered software maintenance.
What this role asks for
Particular to this role
ClickHouse · Golang · Helm · Kafka · OpenTelemetry · Prometheus · Rust · SIEM
Rare in Traversal's other 17 open postings.
Standard for data engineering roles
AWS · Python
Common in data engineering postings at any company.
In most of Traversal's ads
Datadog · ServiceNow
Named in most of its open postings, whatever the role.
84% of this description is written for the role. The rest is Traversal's standard text, folded away below.
Compared with Traversal's other open jobs.
The team: Engineering HIRES STEADILY
This role is one of 6 open in Engineering. 2 were posted in the last 30 days and the oldest in June: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
Datadog (6 of 6), ServiceNow (6 of 6), AWS (5 of 6), Kubernetes (5 of 6), LLM (5 of 6), Python (4 of 6), PostgreSQL (3 of 6)
OTHER ROLES ON THIS TEAM
- AI Engineer - Forward Deployed Engineer posted Nov 3
- Staff Full-Stack Product Engineer posted Aug 10
- AI Engineer - Backend/Platform posted Jul 13
- Cloud Infrastructure Engineer posted Jun 22
- AI Engineer - Product Engineer posted Jun 15
From the team and dates on Traversal's own job board.
Similar jobs
- Staff Software Engineer, Observability/Site Reliability at PsiQuantum
Shares 12 skills: ClickHouse, OpenTelemetry, Helm… · Staff level · same country - Senior Site Reliability Engineer, AI Platforms at Focused
Shares 12 skills: ClickHouse, OpenTelemetry, Helm… · same country - Principal Engineer & Technical Lead — .NET & Azure at Aperia
Shares 11 skills: ClickHouse, OpenTelemetry, Helm… · Staff level · same country - Staff / Principal Platform Engineer at AppGate Cybersecurity, Inc.
Shares 9 skills: OpenTelemetry, Helm, Prometheus… · Staff level · same city - Senior AI Infrastructure Engineer at Hellyeah AI
Shares 11 skills: ClickHouse, OpenTelemetry, Helm… · same country - AI Engineer - Forward Deployed Engineer at Traversal
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.