Despite the growing popularity of AI coding assistants, over 80% of machine learning (ML) projects fail to deliver real business value. This study creates and tests a Machine Learning Canvas, a practical framework that combines business strategy, software engineering, and data science in order to determine the factors that lead to the success of ML projects. We surveyed 150 data scientists and analyzed their responses using statistical modeling. We identified four key success factors: Strategy (clear goals and planning), Process (how work gets done), Ecosystem (tools and infrastructure), and Support (organizational backing and resources). Our results show that these factors are interconnected - each one affects the next. For instance, strong organizational support results in a clearer strategy (β= 0.432, p < 0.001), which improves work processes (β= 0.428, p < 0.001) and builds better infrastructure (β= 0.547, p < 0.001). Together, these elements determine whether a project succeeds. The surprising finding? Although AI assistants make coding faster, they don't guarantee project success. AI assists with the "how" of coding but cannot replace the "why" and "what" of strategic thinking.
mpirical studies suggest that large language models (LLMs) can boost developer productivity by automating certain coding tasks. For example, developers using Copilot finish tasks up to 55% faster and experience a lighter cognitive workload [1]. This is particularly beneficial for less experienced programmers. Anthropic's analysis of approximately 500,000 coding sessions revealed that approximately 79% of interactions with Claude Code involve automation [2]. An empirical study demonstrated increased productivity in Python programming tasks; however, the researchers cautioned that effectiveness depends on task complexity and user expertise [3]. Furthermore, a large-scale field experiment conducted by the Bank for International Settlements revealed a 55% increase in productivity, with LLMs generating approximately one-third of the total lines of code [4]. While these findings affirm productivity improvements, they also imply an ongoing need for human oversight. Code generated by artificial intelligence (AI) addresses repetitive or well-defined programming activities rather than complex architectural decisions [5].
Although LLMs can accelerate code generation and reduce the cognitive burden of routine programming tasks, 1 Martin Prause, martin.prause@xinblue. de . these micro-level efficiency gains do not translate into macro-level project success. In a study of 65 experienced data scientists, it was found that over 80% of AI projects fail -twice the rate of traditional IT projects [6]. The authors identified five primary failure modes: (1) misalignment between technical objectives and business problems, (2) poor data quality and infrastructure, (3) attempting to solve problems that are beyond the current capabilities of AI, (4) insufficient organizational readiness, and (5) inadequate governance structures. These findings confirm earlier research indicating high failure rates in machine learning (ML) deployments [7] [8].
In response to these challenges, machine learning operations (MLOps) have emerged as a discipline that combines software engineering principles with machine learning-specific requirements [9]. A systematic literature review identifies MLOps as encompassing continuous integration and deployment, automated testing and validation, model versioning and governance, and production monitoring and maintenance [10]. Similarly, in another study the authors indicate that adoption success rates vary substantially depending on MLOps maturity. More mature firms demonstrate significantly better capabilities in terms of data management, automated deployments, and continuous integration and delivery [11].
These multifaceted challenges highlight the necessity of structured frameworks for planning, communicating, and executing AI and ML projects. The Business Model Canvas (BMC) approach, popularized by Osterwalder, has proven
The Machine Learning Canvas: Empirical Findings on Why Strategy Matters More Than AI Code Generation
Martin Prause E effective in creating shared mental models among diverse stakeholders [12]. Recent applications to AI contexts include the AI Model Canvas, which adapts Canvas principles to the specific requirements of machine learning [13]. However, existing Canvas frameworks often fail to address the full complexity of ML projects. They typically focus on either technical aspects such as data pipelines and model architecture, or business considerations, such as value propositions and cost structures, without effectively integrating both perspectives. Additionally, these frameworks inadequately address the dynamic nature of ML projects, which require flexible yet structured approaches to iterative experimentation [14].
These limitations are addressed by developing a Machine Learning Canvas (MLC), which integrates organizational theory, software engineering, and data science. An empirical study using a Structural Equation Modeling (SEM) approach was conducted to identify the success determinants of ML projects in development contexts where large language models serve as coding assistants.
A business model is a blueprint that aligns a company’s strategic objectives with its operational execution [15]. It explains how an organization creates, delivers, and captures value [16]. The hierarchical taxonomy identifies four fundamental dimensions of business models: 1. Value Proposition: What value is created, and for whom? 2. Value Architecture: How is value created through organizational capabilities? 3. Value Network: The ecosystem of relationships that enable value creation. 4. Value Finance: The economic model for capturing created value.
The Business Model Canvas operationalizes these dimensions through nine interconnected building blocks arranged on a visual canvas. This approach facilitates iterative development and enables stakeholders to maintain a holistic view while addressing specific components. ML projects share structural similarities with business models in that they focus o
This content is AI-processed based on open access ArXiv data.
