AI Agent Test Automation Engineer (Python)

About the Team We are Quadcode, a fintech company excelling in financial brokerage activities and delivering advanced financial products to our global clientele. Our flagship product, an internal trading platform, is offered as a Software-as-a-Service (SaaS) solution to other brokers. We’re looking for a confident QA Automation Engineer (Python) to join our AI team and focus on testing developer interfaces, applied AI solutions, and complex multi-agent scenarios. You’ll play a key role in ensuring the quality and reliability of our AI agents and developer-facing products in real-world business applications. What We’re Building We’re creating the IDE of the future — a smart development environment powered by cutting-edge LLM agents. It goes beyond just code: our platform helps developers generate images, sound, video, design, and interfaces. This is an AI startup within a stable product company — the perfect combination of innovation and support. We’ve already built an MVP, have real internal users, and a team of skilled engineers. We're currently in internal testing and getting ready for production launch. We operate in GMT+3, with daily team standups at 12:00 GMT+3. Join us to build the next generation of software tools together! Core Responsibilities • Develop automated tests in Python for complex scenarios. • Build mock services and test environments for agent testing. • Integrate tests into AI-specific CI/CD pipelines. • Monitor AI system performance in production. • Design quality metrics and dashboards for AI monitoring. • Document test scenarios and establish best practices. Developer Interfaces Testing • Test SDKs, APIs, and libraries used by developers. • Validate API documentation and code samples. • Build automation for dev tools and CLI utilities. • Test integration workflows and compatibility with various languages/frameworks. • Test IDE plugins and extensions. • Validate data schemas and OpenAPI specifications. Applied AI Testing • Test real-world AI use cases and production scenarios. • Validate model outputs on production data. • Test AI data processing pipelines. • Benchmark AI inference performance. Agent Scenario Testing • Design tests for multi-agent systems. • Validate interactions between AI agents. • Test task chains and agent workflows. • Ensure behaviour constraints and safety. • Test integration with external systems. Requirements • 3+ years in software testing. • Strong Python knowledge: pytest, unittest, mock, asyncio. • Experience testing UI, APIs and SDKs. • Understanding of ML/AI agent concepts. • Experience with REST/GraphQL APIs, WebSockets. • Familiar with performance testing tools (e.g. locust, k6). • Experience with Docker and containerization. • Familiarity with CI/CD and DevOps practices. • Git proficiency. • English level: B1 or higher. • Fluent in Russian (C2). Nice to Have • Understanding of multi-agent system architecture. • Experience with agent frameworks (LangChain, AutoGen, CrewAI). • Working with LLM APIs (OpenAI, Anthropic, local models). • Knowledge of prompt engineering and LLM testing. • Experience with vector databases (Pinecone, Weaviate, ChromaDB). • Familiarity with async/event-driven testing. Bonus Points • Python development experience beyond testing. • Familiarity with ML frameworks (TensorFlow, PyTorch, Hugging Face). • Experience with cloud AI platforms (AWS Bedrock, Azure OpenAI). • Understanding of RAG architectures. What We Offer • Full-time remote work (External Vendor contract). • Competitive compensation (EUR, USD, or crypto). • Flexible working hours. • Influence over the architecture of AI testing systems. • Access to cutting-edge AI tools and platforms. • A team of professionals specializing in AI. • Positive and friendly atmosphere. With a team of 500+ creative and tech-driven people across the globe, Quadcode is the place where ideas turn into impact, where every project is a chance to grow, innovate, and make your mark. Join us today, and let's shape the future of fintech together! Note: All applications will be treated with strict confidence. We thank all applicants for their interest, however, only those candidates selected for interviews will be contacted. #LI-JM1 Apply tot his job Apply To this Job