Automating Agentic AI Reviews with Custom Claude Skills

Bridging the Gap: Automation Meets Governance

The AWS Agentic AI Lens, while a crucial step towards standardizing agentic AI development, was initially a documentation-only asset, posing a significant hurdle for organizations in regulated industries like Smarsh. Their innovative solution of transforming this lens into a structured dataset and building a custom Claude skill to execute reviews is a powerful demonstration of how to operationalize emerging best practices. This approach not only accelerates the review process from days to hours but also ensures consistency and evidence-based assessment by directly integrating with code repositories. The detailed breakdown of the skill's architecture, including lens_data.json, assessment-guide.md, and build_report.py, provides a robust blueprint for others facing similar challenges. The meticulous validation process, using synthetic repositories, further instills confidence in the skill's reliability.

However, a key limitation is that the Agentic AI Lens is not yet a selectable option within the AWS Well-Architected Tool, meaning this automated solution remains a custom implementation rather than a native AWS service. This necessitates upfront development effort for each organization wishing to adopt it. Furthermore, while Claude is powerful, its effectiveness is still dependent on the quality of the input data (the code repository) and the accuracy of the extracted lens information. The article highlights that human reviewers are still essential for validation and prioritization, indicating that full automation of the review process is not yet achievable. The reliance on Anthropic's Claude also introduces a dependency on a third-party AI assistant, which might be a consideration for some organizations with strict vendor lock-in concerns.

Despite these points, the implications are profound. Organizations in regulated sectors, or any enterprise prioritizing robust AI governance, stand to benefit immensely. Developers can leverage this pattern to integrate AI governance checks earlier in the SDLC, improving time-to-market and reducing compliance risks. The technical details showcase a pragmatic application of LLMs for structured data extraction and analysis, moving beyond simple text generation. This approach directly addresses the challenge of rapidly evolving AI best practices by making them executable. It's a compelling example of how custom AI tooling can bridge the gap between emerging standards and current operational capabilities, especially when native tooling lags behind. The Smarsh case study serves as an excellent template for how to proactively manage AI risks in production environments.

Key Points

• AWS Agentic AI Lens, initially documentation-only, was transformed into a structured dataset to enable automated Well-Architected reviews.
• Smarsh built a custom Claude skill to assess agentic AI workloads against 150 best practices, pulling evidence from GitHub repositories.
• The automated review process reduces review time from days to hours, ensuring consistency and evidence-based assessments.
• The skill's architecture involves lens_data.json, assessment-guide.md, and a Python script build_report.py for generating review workbooks.
• Rigorous validation using synthetic repositories was performed to ensure the skill's accuracy and reliability.
• The approach facilitates more targeted and efficient engagement with AWS Solutions Architects for addressing complex gaps.
• This pattern offers a repeatable capability for organizations, especially in regulated industries, to ensure AI governance before production deployment.

---

📖 Source: Automating agentic AI Well-Architected reviews with the AWS Agentic AI Lens