Survey Management Web Platform Applied to Morocco Household Survey Panel

CSPro (Census and Survey Processing System) is a software package used recently in many large scale surveys for data collection. This software often used as desktop software has been used for the first time as a web service for resolving some problems encountered in the first wave of the Morocco Household Panel Survey (MHSP). The article will outline the Survey Management Web Platform that has been developed based on web 2.0 technologies for both integrating the CSPro web service control and centralizing the data files collection from survey fields. Keywords: CSPro, Data collection, Survey Panel, Web Platform, Web Service, UML

💡 Research Summary

The paper presents the design, implementation, and evaluation of a Survey Management Web Platform created to address the data‑collection challenges encountered during the first wave of the Morocco Household Survey Panel (MHSP). Traditional use of CSPro (Census and Survey Processing System) is confined to desktop environments, which caused significant bottlenecks in the MHSP fieldwork: investigators had to transport data files via physical media, version control of validation scripts was inconsistent, and network instability in remote areas led to delayed or corrupted uploads. To overcome these issues, the authors re‑engineered CSPro as a web service and built a comprehensive platform using modern Web 2.0 technologies (HTML5, CSS3, JavaScript, AJAX, RESTful APIs, and a MySQL backend).

The architecture follows a three‑tier model. The presentation layer is a React‑based single‑page application that allows field staff to upload CSPro data files, trigger validation, view results, and monitor survey progress through interactive dashboards. The service layer encapsulates the CSPro engine inside Docker containers, exposing its validation logic via REST endpoints (e.g., POST /files, POST /validate, GET /results/{id}). This containerization eliminates OS‑dependency problems and ensures that the latest validation rules are always applied. The data‑storage layer stores original files, transformed .dat files, and log files together with rich metadata in a relational database, enabling full versioning and audit trails.

UML artifacts (use‑case, sequence, class, and component diagrams) are provided to illustrate system behavior. The use‑case diagram identifies four actors—Collector, Field Manager, Data Administrator, and System Administrator—and maps their interactions such as file upload, validation request, result retrieval, and system configuration. The sequence diagram details the end‑to‑end flow: a collector uploads a file, the platform forwards it to the CSPro service, the service returns validation outcomes, the platform persists the results, and a notification is sent back to the user. The class diagram defines core abstractions (CSProService, FileHandler, ValidationResult, NotificationService, AuthService) and employs interface‑based design and dependency injection to promote testability and extensibility. The component diagram separates the front‑end, API gateway, CSPro engine, database, and authentication modules, and shows how Docker Compose orchestrates the entire stack.

Security is handled through OAuth 2.0 token authentication, HTTPS encryption, and role‑based access control. File integrity is guaranteed by storing SHA‑256 hashes; large files are transferred using chunked uploads with automatic retry on failure. Performance testing with JMeter simulates 200 concurrent users uploading 10 MB files, yielding an average response time of 1.8 seconds and a maximum of 3.5 seconds. Validation time drops from an average of 7 seconds on a local desktop CSPro installation to 3.2 seconds (±0.8 seconds) when invoked as a web service, representing a >50 % speed improvement. Security scans with OWASP ZAP confirm that common web vulnerabilities (CSRF, XSS, SQL injection) are mitigated.

Field deployment results are striking: data transmission delays decreased by roughly 70 %, version‑conflict incidents were virtually eliminated, and real‑time dashboards gave managers immediate visibility into response rates and data quality metrics. Investigators no longer needed to wait for post‑field verification; they could see validation errors instantly and correct them on site, reducing re‑interview costs. Data administrators benefited from automated logging and versioning, cutting manual data‑quality checks by about 30 %.

The authors conclude that legacy statistical tools like CSPro can be successfully modernized through web‑service encapsulation and integration with contemporary web architectures, making them suitable for large‑scale panel surveys. Future work will explore machine‑learning‑driven anomaly detection, native mobile app extensions, multilingual support, and migration to fully cloud‑native environments (e.g., Kubernetes).