Back Cover

A practical guide to using Intelligent Data Analysis to help solve common life problems presented by data analysis tools In Intelligent Data Analysis, the authors have collected insightful knowledge on the complex topics of data abstraction, machine learning, pattern recognition, and specialized statistical analysis. In-depth information has been crafted to be understood by students and professionals alike, providing an overall look at the tools needed to solve problems resulting from automated data collection. Intelligent Data Analysis discusses common issues found in computer-based patient record-keeping, data warehousing tools, and intelligent alarm systems and presents methods of effective and efficient monitoring. The book provides readers with practical problem-solving insights to address typical issues faced in the engineering and medical fields specifically. The text covers a variety of topics within Intelligent Data Analysis (IDA), including: Linear Regression and Decision Trees Interpretation of Black Box Models including Individual Conditional Expectations and Local Interpretable Model-agnostic Explanations An examination of IDA Applications of IDA and Machine Learning Model Interpretability This exhaustive guide provides in-depth information on how IDA can be improved and implemented in engineering and biomedical applications for students, researchers, and professionals developing systems that include IDA technology.

Flap

A practical guide to using Intelligent Data Analysis to help solve common life problems presented by data analysis tools In Intelligent Data Analysis, the authors have collected insightful knowledge on the complex topics of data abstraction, machine learning, pattern recognition, and specialized statistical analysis. In-depth information has been crafted to be understood by students and professionals alike, providing an overall look at the tools needed to solve problems resulting from automated data collection. Intelligent Data Analysis discusses common issues found in computer-based patient record-keeping, data warehousing tools, and intelligent alarm systems and presents methods of effective and efficient monitoring. The book provides readers with practical problem-solving insights to address typical issues faced in the engineering and medical fields specifically. The text covers a variety of topics within Intelligent Data Analysis (IDA), including: Linear Regression and Decision Trees Interpretation of Black Box Models including Individual Conditional Expectations and Local Interpretable Model-agnostic Explanations An examination of IDA Applications of IDA and Machine Learning Model Interpretability This exhaustive guide provides in-depth information on how IDA can be improved and implemented in engineering and biomedical applications for students, researchers, and professionals developing systems that include IDA technology.

Author Biography

Deepak Gupta completed his PhD (CSE) at Dr. APJ Abdul Kalam Technical University, Lucknow, India; his M.E. (CTA) at Delhi College of Engineering, New Delhi, India; and his B.Tech at Guru Gobind Singh Indraprastha University, Delhi, India. He completed his postdoctoral research on the Internet of Things (IoT) at the National Institute of Telecommunications, Ghaziabad, India. He is a guest editor for SCI and SCOPUS and has co-authored 38 books and published 95 research papers. Siddhartha Bhattacharyya, PhD, is a Professor of Computer Science at CHRIST (Deemed to be University), Bengaluru, India. Ashish Khanna received his PhD from the National Institute of Technology, Kurukshetra, India. He completed his M.Tech and B.Tech at Guru Gobind Singh Indraprastha University, Delhi, India in 2004. He has published 100 research papers and co-authored 22 textbooks on engineering. His research includes distributed computing, distributed systems, cloud computing, and opportunistic networks. He completed his postdoctoral research on the Internet of Things (IoT) at the National Institute of Telecommunications, Ghaziabad, India. Kalpna Sagar received her B.Tech from Indira Gandhi Institute of Technology, Guru Gobind Singh Indraprastha University, Delhi, India and her M.Tech from University School of Information, Communication and Technology, Guru Gobind Singh Indraprastha University, Delhi, India. She is currently pursuing her PhD. Her research includes software engineering, human-computer interaction, and data-mining. She has published numerous research papers and is currently an Assistant Professor and Assistant Dean of Academics at KIET Group of Institutions, Dr. APJ Abdul Kalam Technical University, Lucknow, India.

Table of Contents

List of Contributors xix Series Preface xxiii Preface xxv 1 Intelligent Data Analysis: Black Box Versus White Box Modeling 1
Sarthak Gupta, Siddhant Bagga, and Deepak Kumar Sharma 1.1 Introduction 1 1.1.1 Intelligent Data Analysis 1 1.1.2 Applications of IDA and Machine Learning 2 1.1.3 White Box Models Versus Black Box Models 2 1.1.4 Model Interpretability 3 1.2 Interpretation of White Box Models 3 1.2.1 Linear Regression 3 1.2.2 Decision Tree 5 1.3 Interpretation of Black Box Models 7 1.3.1 Partial Dependence Plot 7 1.3.2 Individual Conditional Expectation 9 1.3.3 Accumulated Local Effects 9 1.3.4 Global Surrogate Models 12 1.3.5 Local Interpretable Model-Agnostic Explanations 12 1.3.6 Feature Importance 12 1.4 Issues and Further Challenges 13 1.5 Summary 13 References 14 2 Data: Its Nature and Modern Data Analytical Tools 17
Ravinder Ahuja, Shikhar Asthana, Ayush Ahuja, and Manu Agarwal 2.1 Introduction 17 2.2 Data Types and Various File Formats 18 2.2.1 Structured Data 18 2.2.2 Semi-Structured Data 20 2.2.3 Unstructured Data 20 2.2.4 Need for File Formats 21 2.2.5 Various Types of File Formats 22 2.2.5.1 Comma Separated Values (CSV) 22 2.2.5.2 ZIP 22 2.2.5.3 Plain Text (txt) 23 2.2.5.4 JSON 23 2.2.5.5 XML 23 2.2.5.6 Image Files 24 2.2.5.7 HTML 24 2.3 Overview of Big Data 25 2.3.1 Sources of Big Data 27 2.3.1.1 Media 27 2.3.1.2 The Web 27 2.3.1.3 Cloud 27 2.3.1.4 Internet of Things 27 2.3.1.5 Databases 27 2.3.1.6 Archives 28 2.3.2 Big Data Analytics 28 2.3.2.1 Descriptive Analytics 28 2.3.2.2 Predictive Analytics 28 2.3.2.3 Prescriptive Analytics 29 2.4 Data Analytics Phases 29 2.5 Data Analytical Tools 30 2.5.1 Microsoft Excel 30 2.5.2 Apache Spark 33 2.5.3 Open Refine 34 2.5.4 R Programming 35 2.5.4.1 Advantages of R 36 2.5.4.2 Disadvantages of R 36 2.5.5 Tableau 36 2.5.5.1 How TableauWorks 36 2.5.5.2 Tableau Feature 37 2.5.5.3 Advantages 37 2.5.5.4 Disadvantages 37 2.5.6 Hadoop 37 2.5.6.1 Basic Components of Hadoop 38 2.5.6.2 Benefits 38 2.6 Database Management System for Big Data Analytics 38 2.6.1 Hadoop Distributed File System 38 2.6.2 NoSql 38 2.6.2.1 Categories of NoSql 39 2.7 Challenges in Big Data Analytics 39 2.7.1 Storage of Data 40 2.7.2 Synchronization of Data 40 2.7.3 Security of Data 40 2.7.4 Fewer Professionals 40 2.8 Conclusion 40 References 41 3 Statistical Methods for Intelligent Data Analysis: Introduction and Various Concepts 43
Shubham Kumaram, Samarth Chugh, and Deepak Kumar Sharma 3.1 Introduction 43 3.2 Probability 43 3.2.1 Definitions 43 3.2.1.1 Random Experiments 43 3.2.1.2 Probability 44 3.2.1.3 Probability Axioms 44 3.2.1.4 Conditional Probability 44 3.2.1.5 Independence 44 3.2.1.6 Random Variable 44 3.2.1.7 Probability Distribution 45 3.2.1.8 Expectation 45 3.2.1.9 Variance and Standard Deviation 45 3.2.2 Bayes' Rule 45 3.3 Descriptive Statistics 46 3.3.1 Picture Representation 46 3.3.1.1 Frequency Distribution 46 3.3.1.2 Simple Frequency Distribution 46 3.3.1.3 Grouped Frequency Distribution 46 3.3.1.4 Stem and Leaf Display 46 3.3.1.5 Histogram and Bar Chart 47 3.3.2 Measures of Central Tendency 47 3.3.2.1 Mean 47 3.3.2.2 Median 47 3.3.2.3 Mode 47 3.3.3 Measures of Variability 48 3.3.3.1 Range 48 3.3.3.2 Box Plot 48 3.3.3.3 Variance and Standard Deviation 48 3.3.4 Skewness and Kurtosis 48 3.4 Inferential Statistics 49 3.4.1 Frequentist Inference 49 3.4.1.1 Point Estimation 50 3.4.1.2 Interval Estimation 50 3.4.2 Hypothesis Testing 51 3.4.3 Statistical Significance 51 3.5 Statistical Methods 52 3.5.1 Regression 52 3.5.1.1 Linear Model 52 3.5.1.2 Nonlinear Models 52 3.5.1.3 Generalized Linear Models 53 3.5.1.4 Analysis of Variance 53 3.5.1.5 Multivariate Analysis of Variance 55 3.5.1.6 Log-Linear Models 55 3.5.1.7 Logistic Regression 56 3.5.1.8 Random Effects Model 56 3.5.1.9 Overdispersion 57 3.5.1.10 Hierarchical Models 57 3.5.2 Analysis of Survival Data 57 3.5.3 Principal Component Analysis 58 3.6 Errors 59 3.6.1 Error in Regression 60 3.6.2 Error in Classification 61 3.7 Conclusion 61 References 61 4 Intelligent Data Analysis with Data Mining: Theory and Applications 63
Shivam Bachhety, Ramneek Singhal, and Rachna Jain Objective 63 4.1 Introduction to Data Mining 63 4.1.1 Importance of Intelligent Data Analytics in Business 64 4.1.2 Importance of Intelligent Data Analytics in Health Care 65 4.2 Data and Knowledge 65 4.3 Discovering Knowledge in Data Mining 66 4.3.1 Process Mining 67 4.3.2 Process of Knowledge Discovery 67 4.4 Data Analysis and Data Mining 69 4.5 Data Mining: Issues 69 4.6 Data Mining: Systems and Query Language 71 4.6.1 Data Mining Systems 71 4.6.2 Data Mining Query Language 72 4.7 Data Mining Methods 73 4.7.1 Classification 74 4.7.2 Cluster Analysis 75 4.7.3 Association 75 4.7.4 Decision Tree Induction 76 4.8 Data Exploration 77 4.9 Data Visualization 80 4.10 Probability Concepts for Intelligent Data Analysis (IDA) 83 Reference 83 5 Intelligent Data Analysis: Deep Learning and Visualization 85
Than D. Le and Huy V. Pham 5.1 Introduction 85 5.2 Deep Learning and Visualization 86 5.2.1 Linear and Logistic Regression and Visualization 86 5.2.2 CNN Architecture 89 5.2.2.1 Vanishing Gradient Problem 90 5.2.2.2 Convolutional Neural Networks (CNNs) 91 5.2.3 Reinforcement Learning 91 5.2.4 Inception and ResNet Networks 93 5.2.5 Softmax 94 5.3 Data Processing and Visualization 97 5.3.1 Regularization for Deep Learning and Visualization 98 5.3.1.1 Regularization for Linear Regression 98 5.4 Experiments and Results 102 5.4.1 Mask RCNN Based on Object Detection and Segmentation 102 5.4.2 Deep Matrix Factorization 108 5.4.2.1 Network Visualization 108 5.4.3 Deep Learning and Reinforcement Learning 111 5.5 Conclusion 112 References 113 6 A Systematic Review on the Evolution of Dental Caries Detection Methods and Its Significance in Data Analysis Perspective 115
Soma Datta, Nabendu Chaki, and Biswajit Modak 6.1 Introduction 115 6.1.1 Analysis of Dental Caries 115 6.2 Different Caries Lesion Detection Methods and Data Characterization 119 6.2.1 Point Detection Method 120 6.2.2 Visible Light Property Method 121 6.2.3 Radiographs 121 6.2.4 Light-Emitting Devices 123 6.2.5 Optical Coherent Tomography (OCT) 125 6.2.6 Software Tools 125 6.3 Technical Challenges with the Existing Methods 126 6.3.1 Challenges in Data Analysis Perspective 127 6.4 Result Analysis 129 6.5 Conclusion 129 Acknowledgment 131 References 131 7 Intelligent Data Analysis Using Hadoop Cluster – Inspired MapReduce Framework and Association Rule Mining on Educational Domain 137
Pratiyush Guleria and Manu Sood 7.1 Introduction 137 7.1.1 Research Areas of IDA 138 7.1.2 The Need for IDA in Education 139 7.2 Learning Analytics in Education 139 7.2.1 Role of Web-Enabled and Mobile Computing in Education 141 7.2.2 Benefits of Learning Analytics 142 7.2.3 Future Research Directions of IDA 142 7.3 Motivation 142 7.4 Literature Review 143 7.4.1 Association Rule Mining and Big Data 143 7.5 Intelligent Data Analytical Tools 145 7.6 Intelligent Data Analytics Using MapReduce Framework in an Educational Domain 149 7.6.1 Data Description 149 7.6.2 Objective 150 7.6.3 Proposed Methodology 150 7.6.3.1 Stage 1 Map Reduce Algorithm 150 7.6.3.2 Stage 2 Apriori Algorithm 150 7.7 Results 151 7.8 Conclusion and Future Scope 153 References 153 8 Influence of Green Space on Global Air Quality Monitoring: Data Analysis Using K-Means Clustering Algorithm 157
Gihan S. Pathirana and Malka N. Halgamuge 8.1 Introduction 157 8.2 Material and Methods 159 8.2.1 Data Collection 159 8.2.2 Data Inclusion Criteria 159 8.2.3 Data Preprocessing 159 8.2.4 Data Analysis 161 8.3 Results 161 8.4 Quantitative Analysis 163 8.4.1 K-Means Clustering 163 8.4.2 Level of Difference of Green Area 167 8.5 Discussion 167 8.6 Conclusion 169 References 170 9 IDA with Space Technology and Geographic Information System 173
Bright Keswani, Tarini Ch. Mishra, Ambarish G. Mohapatra, Poonam Keswani, Priyatosh Sahu, and Anish Kumar Sarangi 9.1 Introduction 173 9.1.1 Real-Time in Space 176 9.1.2 Generating Programming Triggers 178 9.1.3 Analytical Architecture 178 9.1.4 Remote Sensing Big Data Acquisition Unit (RSDU) 180 9.1.5 Data Processing Unit 180 9.1.6 Data Analysis and Decision Unit 181 9.1.7 Analysis 181 9.1.8 Incorporating Machine Learning and Artificial Intelligence 181 9.1.8.1 Methodologies Applicable 182 9.1.8.2 Support Vector Machines (SVM) and Cross-Validation 182 9.1.8.3 Massively Parallel Computing and I/O 183 9.1.8.4 Data Architecture and Governance 183 9.1.9 Real-Time Spacecraft Detection 185 9.1.9.1 Active Phased Array 186 9.1.9.2 Relay Communication 186 9.1.9.3 Low-Latency Random Access 186 9.1.9.4 Channel Modeling and Prediction 186 9.2 Geospatial Techniques 187 9.2.1 The Big-GIS 187 9.2.2 Technologies Applied 187 9.2.2.1 Internet of Things and Sensor Web 188 9.2.2.2 Cloud Computing 188 9.2.2.3 Stream Processing 188 9.2.2.4 Big Data Analytics 188 9.2.2.5 Coordinated Observation 188 9.2.2.6 Big Geospatial Data Management 189 9.2.2.7 Parallel Geocomputation Framework 189 9.2.3 Data Collection Using GIS 189 9.2.3.1 NoSQL Databases 190 9.2.3.2 Parallel Processing 190 9.2.3.3 Knowledge Discovery and Intelligent Service 190 9.2.3.4 Data Analysis 191 9.3 Comparative Analysis 192 9.4 Conclusion 192 References 194 10 Application of Intelligent Data Analysis in Intelligent Transportation System Using IoT 199
Rakesh Roshan and Om Prakash Rishi 10.1 Introduction to Intelligent Transportation System (ITS) 199 10.1.1 Working of Intelligent Transportation System 201 10.1.2 Services of Intelligent Transportation System 201 10.1.3 Advantages of Intelligent Transportation System 203 10.2 Issues and Challenges of Intelligent Transportation System (ITS) 204 10.2.1 Communication Technology Used Currently in ITS 205 10.2.2 Challenges in the Implementation of ITS 206 10.2.3 Opportunity for Popularity of Automated/Autonomous/Self-Driving Car or Vehicle 207 10.3 Intelligent Data Analysis Makes an IoT-Based Transportation System Intelligent 208 10.3.1 Introduction to Intelligent Data Analysis 208 10.3.2 How IDA Makes IoT-Based Transportation Systems Intelligent 210 10.3.2.1 Traffic Management Through IoT and Intelligent Data Analysis 210 10.3.2.2 Tracking of Multiple Vehicles 211 10.4 Intelligent Data Analysis for Security in Intelligent Transportation System 212 10.5 Tools to Support IDA in an Intelligent Transportation System 215 References 217 11 Applying Big Data Analytics on Motor Vehicle Collision Predictions in New York City 219
Dhanushka Abeyratne and Malka N. Halgamuge 11.1 Introduction 219 11.1.1 Overview of Big Data Analytics on Motor Vehicle Collision Predictions 219 11.2 Materials and Methods 220 11.2.1 Collection of Raw Data 220 11.2.2 Data Inclusion Criteria 220 11.2.3 Data Preprocessing 220 11.2.4 Data Analysis 221 11.3 Classification Algorithms and K-Fold Validation Using Data Set Obtained from NYPD (2012–2017) 223 11.3.1 Classification Algorithms 223 11.3.1.1 k-Fold Cross-Validation 223 11.3.2 Statistical Analysis 225 11.4 Results 225 11.4.1 Measured Processing Time and Accuracy of Each Classifier 225 11.4.2 Measured p-Value in each Vehicle Group Using K-Means Clustering/One-Way ANOVA 227 11.4.3 Identified High Collision Concentration Locations of Each Vehicle Group 229 11.4.4 Measured Different Criteria for Further Analysis of NYPD Data Set (2012–2017) 229 11.5 Discussion 233 11.6 Conclusion 237 References 238 12 A Smart and Promising Neurological Disorder Diagnostic System: An Amalgamation of Big Data, IoT, and Emerging Computing Techniques 241
Prableen Kaur and Manik Sharma 12.1 Introduction 241 12.1.1 Difference Between Neurological and Psychological Disorders 241 12.2 Statistics of Neurological Disorders 243 12.3 Emerging Computing Techniques 244 12.3.1 Internet of Things 244 12.3.2 Big Data 245 12.3.3 Soft Computing Techniques 245 12.4 Related Works and Publication Trends of Articles 249 12.5 The Need for Neurological Disorders Diagnostic System 251 12.5.1 Design of Smart and Intelligent Neurological Disorders Diagnostic System 251 12.6 Conclusion 259 References 260 13 Comments-Based Analysis of a Bug Report Collection System and Its Applications 265
Arvinder Kaur and Shubhra Goyal 13.1 Introduction 265 13.2 Background 267 13.2.1 Issue Tracking System 267 13.2.2 Bug Report Statistics 267 13.3 Related Work 268 13.3.1 Data Extraction Process 268 13.3.2 Applications of Bug Report Comments 270 13.3.2.1 Bug Summarization 270 13.3.2.2 Emotion Mining 271 13.4 Data Collection Process 272 13.4.1 Steps of Data Extraction 273 13.4.2 Block Diagram for Data Extraction 274 13.4.3 Reports Generated 274 13.4.3.1 Bug Attribute Report 274 13.4.3.2 Long Description Report 275 13.4.3.3 Bug Comments Reports 275 13.4.3.4 Error Report 275 13.5 Analysis of Bug Reports 275 13.5.1 Research Question 1: Is the Performance of Software Affected by Open Bugs that are Critical in Nature? 275 13.5.2 Research Question 2: How Can Test Leads Improve the Performance of Software Systems? 277 13.5.3 Research Question 3: Which Are the Most Error-Prone Areas that Can Cause System Failure? 277 13.5.4 Research Question 4: Which Are the Most Frequent Words and Keywords to Predict Most Critical Bugs? 279 13.5.5 Research Questions 5: What is the Importance of Frequent Words Mined from Bug Reports? 281 13.6 Threats to Validity 284 13.7 Conclusion 284 References 286 14 Sarcasm Detection Algorithms Based on Sentiment Strength 289
Pragya Katyayan and Nisheeth Joshi 14.1 Introduction 289 14.2 Literature Survey 291 14.3 Experiment 294 14.3.1 Data Collection 294 14.3.2 Finding SentiStrengths 294 14.3.3 Proposed Algorithm 295 14.3.4 Explanation of the Algorithms 297 14.3.5 Classification 300 14.3.5.1 Explanation 300 14.3.6 Evaluation 302 14.4 Results and Evaluation 303 14.5 Conclusion 305 References 305 15 SNAP: Social Network Analysis Using Predictive Modeling 307
Samridhi Seth and Rahul Johari 15.1 Introduction 307 15.1.1 Types of Predictive Analytics Models 307 15.1.2 Predictive Analytics Techniques 308 15.1.2.1 Regression Techniques 308 15.1.2.2 Machine Learning Techniques 308 15.2 Literature Survey 309 15.3 Comparative Study 313 15.4 Simulation and Analysis 313 15.4.1 Few Analyses Made on the Data Set Are Given Below 314 15.4.1.1 Duration of Each Contact Was Found 314 15.4.1.2 Total Number of Contacts of Source Node with Destination Node Was Found for all Nodes 314 15.4.1.3 Total Duration of Contact of Source Node with Each Node Was Found 315 15.4.1.4 Mobility Pattern Describes Direction of Contact and Relation Between Number of Contacts and Duration of Contact 315 15.4.1.5 Unidirectional Contact, that is, Only 1 Node is Contacting Second Node but Vice Versa is Not There 317 15.4.1.6 Graphical Representation for the Duration of Contacts with Each Node is Given below 317 15.4.1.7 Rank and Percentile for Number of Contacts with Each Node 320 15.4.1.8 Data Set is Described for Three Days Where Time is Calculated in Seconds. Data Set can be Divided Into Three Days. Some of the Analyses Conducted on the Data set Day Wise Are Given Below 326 15.5 Conclusion and Future Work 329 References 329 16 Intelligent Data Analysis for Medical Applications 333
Moolchand Sharma, Vikas Chaudhary, Prerna Sharma, and R. S. Bhatia 16.1 Introduction 333 16.1.1 IDA (Intelligent Data Analysis) 335 16.1.1.1 Elicitation of Background Knowledge 337 16.1.2 Medical Applications 337 16.2 IDA Needs in Medical Applications 338 16.2.1 Public Health 339 16.2.2 Electronic Health Record 339 16.2.3 Patient Profile Analytics 339 16.2.3.1 Patient's Profile 339 16.3 IDA Methods Classifications 339 16.3.1 Data Abstraction 339 16.3.2 Data Mining Method 340 16.3.3 Temporal Data Mining 341 16.4 Intelligent Decision Support System in Medical Applications 341 16.4.1 Need for Intelligent Decision System (IDS) 342 16.4.2 Understanding Intelligent Decision Support: Some Definitions 342 16.4.3 Advantages/Disadvantages of IDS 344 16.5 Conclusion 345 References 345 17 Bruxism Detection Using Single-Channel C4-A1 on Human Sleep S2 Stage Recording 347
Md Belal Bin Heyat, Dakun Lai, Faijan Akhtar, Mohd Ammar Bin Hayat, Shafan Azad, Shadab Azad, and Shajan Azad 17.1 Introduction 347 17.1.1 Side Effect of Poor Snooze 348 17.2 History of Sleep Disorder 349 17.2.1 Classification of Sleep Disorder 349 17.2.2 Sleep Stages of the Human 351 17.3 Electroencephalogram Signal 351 17.3.1 Electroencephalogram Generation 351 17.3.1.1 Classification of Electroencephalogram Signal 352 17.4 EEG Data Measurement Technique 352 17.4.1 10–20 Electrode Positioning System 352 17.4.1.1 Procedure of Electrode placement 353 17.5 Literature Review 354 17.6 Subjects and Methodology 354 17.6.1 Data Collection 354 17.6.2 Low Pass Filter 355 17.6.3 Hanning Window 355 17.6.4 Welch Method 356 17.7 Data Analysis of the Bruxism and Normal Data Using EEG Signal 356 17.8 Result 358 17.9 Conclusions 361 Acknowledgments 363 References 364 18 Handwriting Analysis for Early Detection of Alzheimer's Disease 369
Rajib Saha, Anirban Mukherjee, Aniruddha Sadhukhan, Anisha Roy, and Manashi De 18.1 Introduction and Background 369 18.2 Proposed Work and Methodology 376 18.3 Results and Discussions 379 18.3.1 Character Segmentation 380 18.4 Conclusion 384 References 385 Index 387

Details