Cloud-Native Survey Automation Platform

Context

Enterprise survey and analytics platform, built greenfield on Azure. Delivered as a two-person build. A colleague built the web application; I owned everything else: infrastructure, backend services, the event pipeline, data modelling, reporting automation, and CI/CD. The platform replaced a business-critical process previously run entirely on spreadsheets.

Problem Statement

The client ran their survey operations on fragmented, spreadsheet-driven workflows. Manual handoffs, version drift, and no single source of truth made the process hard to scale, audit, or evolve. There was no real-time visibility and no consistent behaviour across environments.

Engineering Approach

I replaced the spreadsheets with an event-driven architecture on Azure, integrated directly with the Qualtrics survey platform.

Surveys as data, not documents Survey definitions live as structured data in PostgreSQL: a versioned question bank with blocks and classifications. Publishing a survey generates it in Qualtrics programmatically through a shared API client. Creating or updating a survey is a data change, not a manual build in a vendor UI.

Event-driven response pipeline When a respondent completes a survey, a Qualtrics webhook fires an Azure Function that pulls the full response, archives the raw record, loads it into PostgreSQL, and triggers the dbt transformation that updates the analytics tables. Survey activity flows into analytics as it happens, with no manual steps or file drops anywhere in the chain.

Platform components A shared Python library centralises Qualtrics and database access, so every component speaks to the outside world the same way. On top of the data layer: dbt analytics models and automated reporting that exports PowerBI dashboards to PowerPoint and PDF for stakeholder distribution. A web application for operating the platform was built by my colleague on the same foundation.

Database lifecycle as code Terraform owns the database resource and Flyway owns the schema inside it: every change is versioned and applied automatically during deployment, so each environment promotion runs the same migration set in order, keeping schemas in sync and changes auditable. No manual steps, no one-off scripts.

CI/CD and security Every change moves through a single GitLab CI pipeline covering infrastructure, schema, services, and analytics models together, so a deployment is one coherent unit rather than coordinated manual steps. Development deploys automatically from main; production releases are tagged and gated behind manual approval. CI authenticates to Azure via OIDC with no stored credentials, and services authenticate to each other through managed identities.

Outcome

• Real-time event ingestion: survey activity is processed as it arrives, replacing batch spreadsheet updates.
• Dynamic survey generation: surveys are created and updated programmatically via the Qualtrics API from the event stream and business rules, removing manual template maintenance.
• Automated stakeholder reporting: PowerBI reports are exported to PowerPoint and PDF without manual assembly.
• A single automated pipeline from ingestion to analytics, with consistent infrastructure and schema across environments.
• New event sources and consumers can be added without reworking the core design.

What Made This Complex

Spreadsheet-driven processes encode years of business logic implicitly: in formulas, conventions, and manual judgment calls. Translating that into an explicit event-driven design meant surfacing and formalising rules that had never been written down.

The breadth was the other dimension: one engineer delivering infrastructure, event ingestion, a shared service library, analytics modelling, report automation, and CI/CD as a coherent platform rather than a collection of parts.