Resource usage monitoring for web systems using real-time statistical analysis of log data

Journal article

For Web-based systems accessible from the Internet, it is difficult to estimate workloads precisely. Precise estimation of resources necessary for the system is important for effective utilization of resources in a datacenter. Therefore, capacity planning to forecast the amount of IT resources necessary for a system is important. In capacity planning, the amount of resources necessary for a system is calculated based upon numbers determined by the architecture and business requirements of the system. An example of a number determined by the architecture is the amount of memory required by a single user. An example of a number determined by business requirements is the estimated maximum number of simultaneous users. By multiplying these two numbers, a maximum memory requirement can be calculated. Usually, system memory consumption and the number of simultaneous users are monitored during operation, and if either value exceeds a threshold, an alarm is sent to operators. The authors propose a method to monitor memory consumption per user from memory consumption data and the number of users, and perform statistical significance testing in real time by applying a stream database. The window size used in a CQL statement for the test affects the precision of the test and memory consumption of the stream database. Through experimentation, the authors propose an optimal window size.

Authors: Yoshino, M; Handa, A; Komoda, N; Oba, M

• Journal: Recent Researches in Computational Techniques, Non-Linear Systems and Control - Proc. of the 13th WSEAS Int. Conf. on MAMECTIS'11, NOLASC'11, CONTROL'11, WAMUS'11
• Pages: 216-221
• Publication date: 2011-01-01
• Language: English
• Keywords: Web systems; Statistical testing; Stream database; Resource usage; Capacity planning; Real-time analysis