AI Performance Engineer

Job Title:AI Performance EngineerLocation:Bay Area, CA [ONSITE]
Job Description:
The engineer will lead performance benchmarking, analysis, and tuning of provisioned AI clusters
across distributed training, inference, and GPU collective communication. This senior specialist role
establishes reproducible baselines, identifies gaps against comparable reference configurations, and
works with platform and network teams to resolve bottlenecks. The engineer will validate improvements
while preserving model quality, numerical correctness, and production stability.
Key responsibilities-

• Define benchmark plans and acceptance criteria covering single-node and multi-node workloads.
• Compare results with references using equivalent hardware, models, precision, software versions,
• and workload settings.
• Build automated benchmark suites using MLPerf-style methodologies, NCCL Tests, NeMo/Megatron, TensorRT-LLM/trtllm-bench, NVIDIA AIPerf, and vLLM/SGLang load testing.
• Measure training throughput, step time, model FLOPs utilization (MFU), scaling efficiency, and collective bandwidth and latency. Track inference throughput, time to first token, inter-token and tail latency, and GPU memory use.
• Profile bottlenecks with PyTorch Profiler, Linux perf, NVIDIA Nsight Systems/Compute, and DCGM.
• Correlate workload behavior with compute, memory, storage, network, and host-level evidence.
• Tune batch sizes, sequence lengths, data loading, tensor/data/pipeline parallelism, and communication overlap. Optimize serving concurrency, batching, and KV-cache settings for representative traffic.
• Evaluate supported precision and quantization options against model quality and numerical correctness requirements. Document throughput, latency, memory, and quality tradeoffs before recommending changes.
• Investigate NCCL, GPU interconnect, RDMA, topology, and host configuration issues with platform and network teams. Separate application and framework bottlenecks from infrastructure limitations.
• Control warm-up, repeat runs, report variability, and check for load-generator bottlenecks. Deliver profiling evidence, prioritized remediation, validated configurations, and automated performance regression tests.

Required experience and skills-

• 7 years in performance engineering, HPC, ML systems, or GPU infrastructure, including hands-on benchmarking of distributed training and inference workloads.
• Strong Python and Linux skills for benchmark automation, evidence collection, system diagnostics, and repeatable test execution; practical C /CUDA understanding.
• Experience with PyTorch, GPU profiling, collective communication, and distributed execution, including the impact of parallelism, synchronization, memory use, and data movement.
• Hands-on use of training and inference benchmarking tools, with the ability to select representative workloads and interpret throughput, latency distributions, and scaling behavior.
• Knowledge of benchmark statistics and experimental controls, including warm-up, repeated measurements, run-to-run variation, and fair comparison against reference results.
• Ability to isolate cross-layer bottlenecks, validate performance changes without compromising correctness, and explain findings and remediation priorities to engineering stakeholders.

Preferred experience-

• Experience qualifying NVIDIA AI clusters at scale, including GPU topology, InfiniBand/RoCE, RDMA, NCCL behavior, and hardware, driver, and runtime compatibility.
• Experience with NeMo/Megatron training, MFU analysis, TensorRT-LLM, vLLM/SGLang, NVIDIA Dynamo, and AIPerf or GenAI-Perf.
• Experience integrating benchmark suites with deployment and release validation, using Nsight and
• DCGM evidence to close performance gaps and detect regressions.

Recommended certifications-

• Preferred: NCP-AIO. Specialization: NCP-GENL. Foundational: NCA-AIIO.
• Certifications are preferred credentials. Candidates should demonstrate the corresponding skills
• through practical benchmarking, profiling, and performance tuning experience.

Expected deliverables and success measures-

• Deliver automated benchmark suites, reproducible baseline reports, profiling evidence, prioritized remediation plans, validated tuning configurations, and regression tests. Success is demonstrated through documented closure of gaps against comparable references, improved throughput and scaling or reduced latency, and repeatable results that preserve agreed model quality and numerical correctness.