Methods of Situation Analysis and Graphical Visualisation of Big Data Streams

The article presents an approach to developing a data processing system dealing with manipulation and deep analytics of heterogeneous data streams. We analysed existing instruments for stream data processing. In order to support decision making, we extract the necessary information from message streams to monitor and predict evolution of situations. We base our approach on sequential event detection in a text stream, forming situations and constructing scenarios of their subsequent evolution. We suggest using complex graph models, the metagraph model in particular, in order to represent cause-and-effect and hierarchical connections between events and situations. We consider the issues of developing hardware for acceleration of operations dealing with these models. We employ techniques from so-called "cognitive graphics", in particular dynamic anamorphosis, in order to ensure that the decision maker uses the results of this data stream analysis in a highly efficient manner. We propose a semiotic system structure for processing big data streams

References

[1] Kreps J. Putting Apache Kafka to use: a practical guide to building a stream data platform (part 1). Available at: https://www.confluent.io/blog/stream-data-platform-1 (accessed: 10.02.2017).

[2] Samarev R.S. Review of streaming processing field. Trudy ISP RAN [Proceedings of ISP RAS], 2017, vol. 29, no. 1, pp. 231–260 (in Russ.). DOI: 10.15514/ISPRAS-2017-29(1)-13Available at: http://www.ispras.ru/proceedings/isp_29_2017_1/isp_29_2017_1_231

[3] Andrade H.C.M., Gedik B., Turaga D.S. Fundamentals of stream processing: application design, systems, and analytics. Cambridge University Press, 2014. 558 p.

[4] Chintapalli S., Dagit D., Evans B., Farivar R., Graves T., Holderbaugh M., Liu Z., Nusbaum K., Patil K., Peng B.J., Poulosky P. Benchmarking streaming computation engines: storm, flink and spark streaming. IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2016, pp. 1789–1792. DOI: 10.1109/IPDPSW.2016.138 Available at: http://ieeexplore.ieee.org/document/7530084

[5] Marz N., Warren J. Data: principles and best practices of scalable realtime data systems. Manning Publications, 2015. 328 p.

[6] Defining N-ary relations on the semantic web. W3C Working Group Note 12 April 2006. Available at: http://www.w3.org/TR/swbp-n-aryRelations (accessed: 10.02.2017).

[7] Chernenkiy V.M., Terekhov V.I., Gapanyuk Yu.E. [Representation of complex networks based on metagraphs]. Neyroinformatika — 2016. XVIII Vserossiyskaya nauchno-tekhnicheskaya konferentsiya. Sbornik nauchnykh trudov. Ch. 1 [Neuroinformatics — 2016. Proc. XVIII Russ. Sci.-Tech. Conf. P. 1]. Moscow, MEPhI Publ., 2016. Pp. 225–235 (in Russ.).

[8] Samokhvalov E.N., Revunkov G.I., Gapanyuk Yu.E. Metagraphs for information systems semantics and pragmatics definition. Vestn. Mosk. Gos. Tekh. Univ. im. N.E. Baumana, Priborostr. [Herald of the Bauman Moscow State Tech. Univ., Instrum. Eng.], 2015, no. 1, pp. 83–99 (in Russ.). DOI: 10.18698/0236-3933-2015-1-83-99

[9] Gapanyuk Yu.E., Revunkov G.I., Fedorenko Yu.S. Predicate representation of metagraph data model. Informatsionno-izmeritelnye i upravlyayushchie sistemy [Information-measuring and Control Systems], 2016, no. 12, pp. 122–131 (in Russ.).

[10] Berezkin D.V., Revunkov G.I., Gapanyuk Yu.E. [Approaches to hybrid databank construction for informational system]. Gibridnye i sinergeticheskie intellektualnye sistemy. Materialy III Vserossiyskoy Pospelovskoy konferentsii s mezhdunarodnym uchastiem [Hybrid and synergetic intelligence systems. Proc. III Russ. Pospelov Conf. with International Participation]. Kaliningrad, I. Kant BFU Publ., 2016. Pp. 141–148 (in Russ.).

[11] Macko P., Margo D., Seltzer M. Performance introspection of graph databases. Proc. 6th Int. Systems and Storage Conf. (SYSTOR 13). ACM, 2013. 10 p. DOI: 10.1145/2485732.2485750 Available at: https://dl.acm.org/citation.cfm?doid=2485732.2485750

[12] Basu A., Gandhi J., Chang J., Hill M.D., Swift M.M. Efficient virtual memory for big memory servers. SIGARCH Comput. Archit. News, 2013, vol. 41, no. 3, pp. 237–248. DOI: 10.1145/2508148.2485943 Available at: https://dl.acm.org/citation.cfm?id=2485943

[13] Delorimier M., Kapre N., Mehta N., Dehon A. Spatial hardware implementation for sparse graph algorithms in GraphStep. ACM Trans. Auton. Adapt. Syst., 2011, vol. 6, no. 3, art. 17.

[14] Mitsuishi T., Nomura Sh., Suzuki J., Hayashi Yu., Kan M., Amano H. Accelerating breadth first search on GPU-BOX. SIGARCH Comput. Archit. News, 2014, vol. 42, no. 4, pp. 81–86. DOI: 10.1145/2693714.2693729 Available at: https://dl.acm.org/citation.cfm?doid=2693714.2693729

[15] Gulati K., Paul S., Khatri S.P., Patil S., Jas A. FPGA-based hardware acceleration for boolean satisfiability. ACM Trans. Des. Autom. Elect. Syst., 2009, vol. 14, no. 2. DOI: 10.1145/1497561.1497576 Available at: https://dl.acm.org/citation.cfm?id=1497576

[16] Kumar Sn., Shriraman A., Srinivasan V., Lin D., Phillips J. SQRL: hardware accelerator for collecting software data structures. Proc. 23rd Int. Conf. on Parallel Architectures and Compilation (PACT14), ACM, 2014, pp. 475–476. DOI: 10.1145/2628071.2628118 Available at: https://dl.acm.org/citation.cfm?id=2628118

[17] Popov A. An introduction to the MISD technology. Proc. 50th Hawaii Int. Conf. on System Sciences (HICSS50), 2017, pp. 1003–1012.

[18] Andreev A.M., Berezkin D.V., Kozlov I.A. Automated topic monitoring based on event detection in text stream. Informatsionno-izmeritelnye i upravlyayushchie sistemy [Information-measuring and Control Systems], 2017, vol. 15, no. 3, pp. 49–60 (in Russ.).

[19] Andreev A.M., Berezkin D.V., Kozlov I.A. Podkhod k avtomatizirovannomu monitoringu i prognozirovaniyu razvitiya innovatsionnykh obrazovatelnykh tekhnologiy. Nauka i obrazovanie: nauchnoe izdanie [Science and Education: Scientific Publication], 2016, no. 7, pp. 196–208 (in Russ.). Available at: http://technomag.edu.ru/jour/article/view/999

[20] Guseyn-zade S.M., Tikunov V.S. Anamorfozy: chto eto takoe? [Anamorphosis: what is this?] Moscow, Editorial URSS Publ., 1999. 168 p.

[21] Terekhov V.I. Primenenie kognitivnoy kompyuternoy grafiki v sistemakh podderzhki prinyatiya resheniya dolzhnostnykh lits organov voennogo upravleniya [Application of cognitive computer graphics in decision support systems of military administration official bodies]. Moscow, Obshchevoyskovaya akademiya VS RF Publ., 2012. 150 p.

[22] Terekhov V.I. Application of cognitive computer graphics for urgent data visualization by decision-making persons. Vestn. Mosk. Gos. Tekh. Univ. im. N.E. Baumana, Priborostr. [Herald of the Bauman Moscow State Tech. Univ., Instrum. Eng.], 2012, spec. iss, pp. 109–119 (in Russ.).

[23] Koza J. Genetic programming: on the programming of computers by means of natural selection. MIT Press, 1992. 840 p.