About Questi
Questi is an innovative online platform designed for teachers to facilitate the management of their day-to-day tasks (calendar with custom lesson preparations, customisable report cards with an extensive set of possible evaluation types, and student development tracking) as well as for schools to nurture communication between teachers and parents. The platform offers a user-friendly solution that simplifies communication and encourages collaboration between teachers and students. With Questi, schools can expect to experience an intuitive and streamlined approach to communication, making it easier for teachers to keep parents informed about their child’s progress and stay involved in their child’s education.
The problem
Questi wanted to optimise the API Server logging capabilities and increase the overall platform performance within their app, ensuring seamless and efficient communication between teachers, students, and parents.
Many companies tend to rely too much on their internal team, thinking that they can do everything with the knowledge they have and that they should not look for other solutions. However, I believe that they should do the opposite. By looking at other solutions, they can have a wider range of options to choose from. This is what we did with Devoteam G Cloud.
The goal
To tackle the challenge, Questi wanted to upgrade its logging system. Originally a Cloud Task was triggered for each log, which started an App Engine instance to convert the log to a SQL query and insert it into Questi’s logging database. Questi wanted a more advanced, easier to analyse and cost-effective solution utilising the latest Google Cloud technologies. This modernisation effort aimed to future-proof the logging system while leveraging Questi’s existing Google Cloud Platform environment. This would also enable future business use cases involving ML & AI.
The methodology
Finding a new solution
The initial plan was to adapt the Questi API server so that instead of triggering Cloud Tasks, logs would be written to the Cloud Logging API using the client library. A Logging Sink with an inclusion filter for production logs would be set up, with a destination of a BigQuery dataset. However, since the logs were written in JSON format and had a variable nature that posed challenges for effective handling in BigQuery, it was found unsuitable for Questi’s specific use case.
To fit better with the requirements, we proposed together with Google Cloud to use Log Analytics instead of the above solution. We also set up a linked dataset in BigQuery. With this solution, the logs are being ingested and retained by Cloud Logging, with the Logs Bucket where production logs are being routed to upgraded to Log Analytics. Thanks to Log Analytics and the linked BigQuery dataset, Questi can now fully leverage SQL and BigQuery analytics for its production logs.
Migrating old logs to the new system
Once the setup had been completed and the application logs were saved in Cloud Logging, the experts migrated the old logs as well.
To export the logs from the old Cloud SQL database, they used the export to CSV function of Cloud SQL. Once all data was exported to a Cloud Storage bucket, it could not be inserted into Cloud Logging without first converting the CSV data back to the native log format.
We used a custom tool written in Golang, to transform the CSV data and insert it into Cloud Logging using the Cloud Logging library for Go.
Due to the large amount of data, the migration would take a few hours. With the long-running nature of this migration, we only had two options of where we could execute the tool, GCP Batch or Compute Engine. Since this was a one-time execution, we opted for the latter, Compute Engine.
The new solution is four times cheaper than the old one, which is a significant savings for our company. Devoteam G Cloud provided us with training, which helped us get up and running quickly. They are excellent in support throughout the project. The new solution is future-proof for analytics, which means that we can use it to analyse our data in the future to make better decisions.
The Result
As a result of this modernisation project, Questi can now view the logs that their platform generates much more efficiently, in Cloud Logging. Before the modernisation, they had to write SQL queries and query their database, just to debug a small problem, while the developers can now stream their incoming logs without any effort.
The primary reason for doing this modernisation was to optimise cost.
The ingestion cost has dramatically decreased since no Cloud Tasks or App Engine instances are needed anymore. The ingestion of logs into Cloud Logging is free, and Questi saves money each month.
On the other hand, there is the cost of storing this data. With Cloud Logging being much cheaper than Cloud SQL, the projected cost will be 4 times cheaper.
With this solution in place, it will also be easier for Questi to add additional logging in the future with a very minimal setup and the possibility of leveraging the benefits of Log Analytics.
Get in touch with us and modernise your logging system today!
Benefit from the expertise of a recognised Google Cloud Premier Managed Services Partner. From strong change management to technical excellence and an innovative mindset: Devoteam experts have what it takes to inspire you and make your projects successful.
