A research on clustering and identifying automated communication in the HTTP environment

A lot of HTTP traffics are unnoticed to users because they are automatically generated from software. This caused by HTTP protocol characteristics. For the purpose of communication with servers, HTTP-based applications always automatically and actively send requests to their hosts because HTTPs are designed as connectionless protocols. In addition, all kinds of HTTP communications from software such as a bot, adware, and normal web accesses are mixed clearly. This raises the requirement for clarification of HTTP traffics. Most previous studies concentrated on HTTP-based malicious bot traffics, however, graywares such as adware or unauthorized applications are also becoming serious internal threats since they can stealth sensitive information or web usage experiences from infected systems. In this study, a new method for clustering and identifying HTTP communications is proposed. It focuses on analyzing of HTTP-based software Internet access behaviors. The method is tested with real outbound HTTP communication of a private network. Examination showed improved results with an accuracy rate of 91.18% in clustering and identifying HTTP automated communications. © Springer Nature Singapore Pte Ltd. 2020.