IGAISE

While Generative AI is rapidly becoming central to developer workflows, assisting with tasks from code generation to documentation, organizations face a critical challenge: systematically measuring its true impact. Despite perceived benefits like improved code quality and faster code reviews, some organizations have surprisingly reported negative effects on software delivery performance after adopting AI tools. This apparent contradiction underscores the complexity of assessing GenAI's influence, highlighting the need for a pragmatic and context-aware evaluation framework to ensure real value and avoid missteps.

fortiss and Siemens AG addressed this challenge by developing a comprehensive framework offering actionable guidance and conceptual clarity, enabling the reflect on and assessment of GenAI's impact across various dimensions, including speed, automation, flow, quality, maintainability, and relevance. The solution is grounded in extensive literature analysis, real-world experiences from research and transfer projects, and continuous feedback from industrial partners.

Development of a practical Handbook for Measuring the Efficiency and Effectiveness of Generative AI in Software Engineering, offering context-aware evaluation.

• Provision of guidance, concepts, and recommendations to systematically assess GenAI's impact on software engineering efficiency and effectiveness.
• Enhanced understanding of how GenAI influences various phases of the Software Development Life Cycle (SDLC), from planning and analysis to implementation, testing, and maintenance.
• Clarification of key terminology and contextual factors essential for interpreting GenAI's application in diverse software engineering scenarios

01.09.2024 – 31.03.2025