AI Operations Specialist - LLM & Conversation Systems

AI Operations Specialist – LLM & Conversation Systems

Position Type: Full-Time, Remote

Working Hours: U.S. Business Hours

About the Role

We are looking for a hands-on AI Operations Specialist to own and improve our production AI conversation systems.

We are an AI-powered SMS lead recovery platform serving telehealth brands, processing millions of messages each month. Our AI systems interact directly with customers, and their performance directly impacts client conversions.

In this role, you'll onboard new clients, configure AI conversation systems, monitor live conversations, diagnose issues, improve prompts and system behavior, and measure whether your changes actually improve conversion.

This is not a theoretical AI role. We're looking for someone who has run LLM systems in production and understands how to continuously monitor, troubleshoot, test, and improve them.

Responsibilities

Client Onboarding & AI Setup

• Onboard new clients into the AI conversation platform.

• Set up webhook integrations and client knowledge bases.

• Configure campaigns and conversation workflows.

• Ensure new client systems are properly tested before going live.

• Troubleshoot integration and configuration issues during onboarding.

Production AI Operations

• Monitor live AI-generated conversations and customer interactions.

• Review conversations to identify errors, inconsistencies, and missed opportunities.

• Diagnose AI failure modes and determine their root causes.

• Identify issues affecting conversation quality and conversion.

• Escalate technical issues when necessary and follow through until resolved.

Prompt & System Optimization

• Write, edit, and improve prompts that generate customer-facing messages.

• Adjust system instructions and conversation logic based on real-world performance.

• Test changes systematically before and after deployment.

• Maintain documentation of prompt and system changes.

• Continuously improve AI behavior while maintaining accuracy and consistency.

Performance & Conversion Analysis

• Analyze conversation transcripts and performance data.

• Measure whether system and prompt changes improve conversion.

• Identify patterns across conversations that can inform future improvements.

• Track key performance outcomes and communicate findings to leadership.

• Use data and real customer interactions to guide optimization decisions.

Founder Collaboration

• Work directly with the founder on daily AI operations and improvements.

• Communicate issues, findings, and recommendations clearly.

• Take ownership of problems from identification through resolution.

• Move quickly while maintaining high standards for customer-facing AI output.

Required Experience & Skills

Must-Have

• Proven experience running LLM or AI systems in production .

• Hands-on experience monitoring and improving production AI outputs.

• Experience with prompt engineering and system prompt optimization.

• Strong understanding of LLM behavior, failure modes, and iterative testing.

• Experience analyzing AI-generated conversations or customer-facing outputs.

• Ability to work with webhooks, APIs, knowledge bases, or similar integrations.

• Strong analytical and problem-solving skills.

• Excellent written English — required for editing customer-facing AI messages and documenting system changes.

• Strong attention to detail.

• Ability to work independently and take ownership of production systems.

What Makes You a Strong Fit

• You've actually operated AI systems in production rather than only experimenting with ChatGPT.

• You are comfortable reviewing hundreds or thousands of AI interactions to identify patterns and problems.

• You think in terms of outcomes and conversion , not just whether the AI response sounds good.

• You are methodical about testing changes and measuring their impact.

• You can quickly identify why an AI system is behaving incorrectly and determine how to improve it.

• You communicate clearly and document your work thoroughly.

• You enjoy working directly with a founder in a fast-moving environment.

What Does a Typical Day Look Like?

You'll begin by reviewing the performance of live AI conversations and looking for unusual patterns, failures, or opportunities for improvement.

You may spend part of the day onboarding a new telehealth client, configuring their knowledge base and integrations, and setting up their campaigns.

You'll also review conversation transcripts, identify issues with AI responses, update prompts or system instructions, test your changes, and monitor the resulting conversion performance.

Throughout the day, you'll work directly with the founder to discuss findings, prioritize improvements, and make sure production systems continue performing effectively.

Interview Process

• Initial Recruiter Screening

• Client Interview

• Offer Stage

#LI-AG1