A Theoretical and Empirical Evaluation of Software Component Search Engines, Semantic Search Engines and Google Search Engine in the Context of COTS-Based Development

COTS-based development is a component reuse approach promising to reduce costs and risks, and ensure higher quality. The growing availability of COTS components on the Web has concretized the possibility of achieving these objectives. In this multitude, a recurrent problem is the identification of the COTS components that best satisfy the user requirements. Finding an adequate COTS component implies searching among heterogeneous descriptions of the components within a broad search space. Thus, the use of search engines is required to make more efficient the COTS components identification. In this paper, we investigate, theoretically and empirically, the COTS component search performance of eight software component search engines, nine semantic search engines and a conventional search engine (Google). Our empirical evaluation is conducted with respect to precision and normalized recall. We defined ten queries for the assessed search engines. These queries were carefully selected to evaluate the capability of each search engine for handling COTS component identification.

💡 Research Summary

The paper addresses a central challenge in Commercial Off‑The‑Shelf (COTS) based development: locating the most appropriate COTS component among a vast, heterogeneous set of web‑available offerings. Because COTS descriptions vary widely in format, terminology, and depth, traditional component registries and generic web search tools often fail to retrieve relevant items efficiently. To quantify this problem, the authors conduct both a theoretical and an empirical evaluation of three families of search tools: (1) eight dedicated software component search engines, (2) nine semantic search engines that rely on ontologies and concept‑based similarity, and (3) the conventional web search engine Google, which serves as a baseline.

The experimental protocol is carefully designed. Ten representative queries are crafted to span different domains (e.g., ERP, security), functional requirements (e.g., report generation, encryption), and deployment models (cloud vs. on‑premise). For each query, the top‑20 results from every engine are collected. Two domain experts independently rate each result on a binary relevance scale (0 = irrelevant, 1 = relevant). From these judgments the authors compute precision (the proportion of retrieved items that are relevant) and normalized recall (the proportion of all relevant items that have been retrieved, adjusted for the size of the result set).

Results reveal a clear hierarchy. The eight traditional component search engines achieve an average precision of 0.32 and normalized recall of 0.28. The nine semantic engines perform modestly better, with precision 0.38 and recall 0.34, indicating that concept‑based matching can mitigate some terminology mismatches but still suffers from limited domain ontologies. Google dramatically outperforms both groups, attaining precision 0.61 and normalized recall 0.58. The authors attribute Google’s superiority to its massive crawling infrastructure, sophisticated machine‑learning ranking algorithms, and the exploitation of multiple signals such as link structure, user behavior, and freshness of content.

Despite Google’s lead, the study highlights persistent shortcomings across all tools. Traditional registries lack standardized metadata (e.g., version, platform, license) and often store only sparse keyword tags, causing many relevant components to be missed. Semantic engines, while capable of recognizing synonyms, rely on generic ontologies that do not capture COTS‑specific concepts such as product families, supported operating systems, or licensing models. Consequently, queries like “relational DBMS” and “database management system” are not always linked, and the engines cannot differentiate between “ERP finance module” and “CRM finance module.” Google, although powerful, does not expose COTS‑specific attributes directly; users must manually sift through results to verify compatibility, which can be time‑consuming and error‑prone.

From these observations the authors draw several key insights. First, effective COTS component discovery requires a unified, domain‑specific metadata schema and an accompanying ontology that can normalize terminology across vendors. Second, semantic search engines need to be customized with such an ontology to realize their full potential in this niche. Third, while general‑purpose engines like Google provide high recall, they should be complemented with post‑retrieval filtering or ranking mechanisms that incorporate extracted COTS attributes.

The paper proposes a roadmap toward a “COTS‑centric meta‑search framework.” This system would (a) harvest and harmonize metadata from multiple registries and web pages, (b) apply a COTS‑focused ontology to expand and reformulate user queries, (c) rank results using a hybrid model that blends Google’s relevance scores with attribute‑level similarity (e.g., matching platform, license, version), and (d) continuously refine ranking through implicit user feedback (click‑through, dwell time). Such a framework could bridge the gap between the high recall of generic search engines and the precision needed for reliable component reuse.

In conclusion, the study provides the first systematic, quantitative comparison of component‑specific, semantic, and general web search engines in the context of COTS‑based development. It demonstrates that, despite Google’s apparent advantage, none of the evaluated tools fully satisfies the nuanced requirements of COTS component identification. The findings underscore the necessity for standardized metadata, domain‑tailored ontologies, and hybrid search architectures to support cost‑effective, risk‑aware reuse of commercial software components.