Sean Kim
Project List

Project Archive 05 · Data Engineering · Web Behavior Analytics

Redesigning the Architecture of VODA, a Web Behavior Analytics Service

Click and scroll collection → SQS message processing → Elasticsearch storage → periodic aggregation → heatmaps and charts

PythonSeleniumDockerElasticsearchMySQLAWS SQS

Pre-computing aggregates fixed the timeouts on the analytics screens

VODA is an analytics service that collects page views, clicks and scrolls from website visitors and shows them as traffic charts and heatmaps. As more sites adopted it, the existing architecture could no longer keep up, so a four-person TFT redesigned it. I was responsible for event collection, message processing, aggregation and page capture.

  • 5 casesCollection and aggregation problems I solved myself
  • 4Components I built
  • 4 peopleMembers of the architecture redesign TFT
Mouse-move heatmap screen

Shows where visitors' mice lingered, overlaid on the page. Coordinate collection and aggregation produce the data for this screen. The screenshot comes from the service's introduction material; the company name has been masked.

Translated reference · View Korean original
Assigned role
I fixed click-coordinate collection and built the processing that turns SQS messages into Elasticsearch documents, a refiner that prepares aggregates in advance, and a crawler that captures full pages.
Results
I corrected click coordinates that varied from device to device and stopped messages from being lost or duplicated. The analytics screens now read results the refiner has already prepared instead of aggregating in Elasticsearch every time, which fixed the timeouts.
  1. 1Coordinate collectionRecords coordinates relative to the topmost element under the mouse
  2. 2Message processingShapes SQS messages into the Elasticsearch document format
  3. 3Data refinerAggregates periodically and stores the results in RDS tables
  4. 4Page crawlerCaptures whole pages, even on full-page scrolling sites

Old and new architecture

Before

  1. Web AgentCollects visitors' page views, clicks and scrolls

  2. voda-serverOne server stored collected data in MySQL and also handled refining and screen responses

  3. Analytics screensTraffic charts and heatmaps

AfterParts I built

  1. Collection

    • Web AgentCollects visitors' page views, clicks and scrollsImproved how coordinates are recorded
    • API Gateway, SQSHolds mouse and scroll events in a queue
    • pv-recorder, mouse-recorderStores messages as Elasticsearch documentsMessage shaping, parallel collection, de-duplication
    • page-crawlerCollects page screenshots and element positionsFull-page capture

  2. Storage

    • ElasticsearchEvent logs and aggregates
    • MySQLClient accounts, page data, reports

  3. Refining

    • Aggregation refinerAggregates mouse and scroll data periodicallyBuilt the refiner
    • forecastForecasts visit trends

  4. Visualization

    • presenterAPI that serves data to the analytics screensReturns aggregated results
    • voda-analyticsAnalytics screens for administrators

Expected changes from this technology

Collection that doesn't miss events

With the collector split across several containers, queued messages are picked up in time, and a message that arrives twice is still stored only once.

Heatmaps that are accurate on any device

Coordinates are recorded relative to web elements, so clicks on the same button land in the same spot whatever the screen size.

Analytics screens without the wait

Heavy aggregation is done ahead of time, so administrators can open the screens and see charts and heatmaps without timeouts.

Problem and solution

  1. ProblemClicks were recorded as absolute page coordinates, so the same button ended up at different positions depending on screen size.

    ResolveI found the topmost web element under the mouse and recorded the coordinates within that element.

  2. ProblemAggregating with Elasticsearch Aggregation every time an analytics screen opened made responses so slow that they timed out.

    ResolveI built a new refiner that aggregates periodically and saves the results to RDS tables, so the screens only read finished aggregates.

  3. ProblemWith the section-by-section full-page scrolling common on public-sector sites, a normal capture took only the first section.

    ResolveInstead of the Selenium API, I sent requests directly to Chrome, extended the capture height to the full scroll length and captured in mobile mode.

  4. ProblemIf SQS messages were not picked up for a minute, the queue went over its limit and data was lost.

    ResolveUsing an asynchronous task queue, I split collection commands across several Docker containers so messages are fetched in parallel.

  5. ProblemOnce fetching ran in parallel, the same message was sometimes fetched twice and logs were duplicated.

    ResolveI built the document ID from the message time and user ID, so a repeated message overwrites the existing one.

Analytics service screens

User count analysis screen

Shows visit trends by hour and day of the week along with forecasts. The screenshot comes from the service's introduction material; the company name has been masked.

Translated reference · View Korean original