The system identifies operational failures that might otherwise go unnoticed for weeks—even when the server itself appears to be functioning normally.
Monitors 13 server components across three services, alongside actual operational outcomes such as invoice transmissions and notification messages. We developed an operational tool that aggregates errors from server APIs and user-side PC applications, enabling a workflow that transitions from Slack notifications to root-cause analysis within an administrative dashboard.
- 0.5 seconds → 0.1 secondsAverage response time of the Order Service database, observed immediately after improving slow queries
- 6 casesOperational issues identified and resolved via monitoring alerts
- 13 placesServer components checked every minute
This is a custom-developed administrator status screen. It displays the current status and operational history for each service. Internal identification information has been masked.
Translated reference · View Korean original- Assigned role
- Designed monitoring criteria and notification workflows. I developed everything from monitoring programs and server status dashboards to tools for verifying operational results, collecting API and PC application errors, and an administrator console to aggregate and view all this information.
- Results
- We established a monitoring system that tracks server status, actual operational outcomes, and errors from both the server API and the PC application. This system integrates alerts, recovery notifications, and periodic reports with an administrator console. Through this monitoring, we identified and resolved six operational issues, including slow query performance and problems with invoice and notification message (KakaoTalk) integrations.
- 1Server Monitoring13 locations were judged to be normal, poor performance, faulty, or unconfirmable.
- 2Check task resultsDetermined based on whether the request was actually processed, rather than just whether it was received.
- 3Notifications and reportsAlert when three consecutive failures occur, once more during recovery, and regular reporting twice a day.
- 4Error collectionAggregating and grouping errors from both the server API and the PC application.
Expected changes from this technology
Operational response for a small team
Instead of constantly monitoring multiple management screens, the person in charge receives notifications for anomalies and recovery—reducing the burden of repetitive checks and allowing them to focus on identifying root causes.
Detecting issues before they reach the customer
By detecting failures such as missing delivery information or operational notifications—even when the server is operating normally—we can take corrective action before customers even reach out with inquiries.
Error logs leading to improvements
Consolidates errors occurring on both the server and the user's PC to identify recurring issues, while maintaining an operational record that allows for the comparison of states before and after fixes.
Problem and solution
ProblemEven when the server is functioning normally, processes such as invoice transmission or the sending of notification messages (KakaoTalk) may still fail.
ResolveDistinguished between request receipt and actual processing results, incorporating task failures and incomplete executions into the monitoring system.
ProblemSituations such as the system performing administrative analysis or awaiting initial results were being misinterpreted as actual system failures.
ResolveDistinguishes the source of operations and categorizes items into "pending collection," "results unverified," and "finalization failed," allowing for the identification of issues that require attention.
ProblemIf the data collection connection is lost during a failure, the status history for that period will be empty.
ResolveWe implemented a system to temporarily store observations and transmit them—without duplication—once the connection is restored, allowing for a continuous view of the failure and recovery sequence.
ProblemIf the system waits for an error report when the server is slow or disconnected, both the editing and output processes in the "Furniture Builder" (Gagu Builder) come to a halt.
ResolveErrors are reported separately without interrupting the editing or output process; repeated reports of the same error have been limited.
ProblemError messages may contain tokens or file paths.
ResolveOnly the first line with sensitive parts erased on both the PC and server is left, and the drawing content is not sent.
Confirmed results
- We cross-referenced the administrator status screen, Slack reports, and the history of failures and recoveries.
- Operational graphs show that, following the fix for slow queries, the average database response time shifted from the 0.5-second range to approximately 0.1 seconds.
- Six instances were recorded where the root cause was identified and addressed following alerts from monitoring systems (such as those for invoices, KakaoTalk notifications, or database connections).
13 locationsCheck [Object/Item] every minute
Design and manufacturing platform8 locations
- Nginx (operations, development)Front-end server self-response
- API server (operations, development)Status response and engine loading
- GPU worker (operations, development)Last heartbeat
- DB (operations, development)Attachment and slow lookup
Furniture ordering service3 locations
- Web service (Operations)Initial screen and app file response
- DB (Operations)Attachment and slow lookup
- DB function call (Operations)Average response time per function
Factory management service.2 locations
- Web service (Operations)Login screen and app file response
- API server (Operations)Response to read-only request
- Regular Operation
- Post-payment processing
- Task queue
- External Integration
- Send notification message
Errors are collected as they occur
Server APIDesign and manufacturing platform
5xx and all unhandled exceptions, 4xx only samples
API error screenIdentical errors are grouped together, with their processing status and history tracked.
User PC applicationFurniture Builder, Connector, and Plotter
Transmit only the location and target of the error without interrupting the operation
Local app issues screenTrends by Program, Grouping by Error Code, Marking as Processed
Actual issues detected by the monitoring system 6 cases
The orange line represents the order service database. On the evening we began recording, the average response time dropped from the 0.5-second range to around 0.1 seconds, immediately following a fix for slow queries.
Translated reference · View Korean originalOn the first day the monitoring system was activated, it revealed that the average response time for the order service database was hovering around 0.5 seconds, and that queries for trend data over specific periods in the management console were taking 4–5 seconds on average.
Immediately after fixing the query, the average database response time dropped to around 0.1 seconds. Seventeen other queries with the same root cause were also fixed in the same way.
A deployment check task accessed the file repository while keeping a database transaction open, holding the connection for over 10 seconds.
Only DB searches were done within the connection, and file checking was done after the connection was closed.
Two invoice transmission attempts, initiated at the same time, were both recorded as failures.
Eliminated duplicate transmissions of invoices, such as those caused by transmission conflicts.
The delivery status results for four notification messages never arrived. A review of the last 120 days revealed 65 messages for which no result was received.
We implemented a system that re-queries the status of messages every minute if no result is received after sending.
One order had been failing every time the full invoice was transmitted, a recurring issue over a six-week period starting in early September.
Updated to mark as complete by comparing against already registered numbers.
An attempt to resend an "order complete" notification (via KakaoTalk) upon a virtual account deposit was rejected.
Prevented the "notification talk" (KakaoTalk message) that was already sent at the time of order from being sent again upon payment.
Screen for operational monitoring
Reports the status of all components twice a day. Failure and recovery alerts are also delivered through the same notification stream.
Translated reference · View Korean originalCompare transient spikes with persistent anomalies.
Translated reference · View Korean originalRecords the period from the onset of a failure until recovery.
Translated reference · View Korean original