Big Data

 Application of Big Data Techniques to a Problem From the previous posts, it is clear that Big Data is collected, stored and utilised in the real world at enormous scales by huge companies. Now I want to show directly how these companies use this data to address their problems in the modern world. Netflix Netflix is known for its great, vast choice of shows that users love. But how does Netflix know what each individual customer might want to watch? Netflix recommends specific titles for each user on the home page. It uses multiple algorithms for personalised rankings, searches, similarity ratings, ranking and more to narrow down the shows to display on the recommended screen for each individual customer. It looks at what titles the user has watched, searched for, and rated good or bad so the system can show them similar content. Additionally, if user A shows the same interests as user B, Netflix will show user A the user B's recommended shows. Each show is also put into some of th...

 Types of Visualisation Today, we covered how data can be presented visually. I have spoken about how data is used to draw conclusions and make predictions, however, I haven't yet discussed how data can be visualised to draw these conclusions. It would be impossible to understand vast data through numerical spreadsheets of data or even just large stores of unorganised data. This is why it's important to use data visualisation. Data visualisation is the graphical representation of data. We can use charts, graphs, maps, and other visualisation tools to see and understand trends, patterns and outliers. Some examples are: Tables These are a collection of related data organised in columns and rows. Graphs These are diagrams that show the relationship between variables across two axes. Maps These are diagrams to display location data in their geographic locations. Infographics These are representations of information or data in a graphic format to make the data easily understandable....

 Data Mining Techniques I have previously spoken about data analysis and the real world impact of it. Today, I discovered a website which highlighted the common data mining techniques which are used to discover hidden patterns and correlations is vast amounts of data, which is then used for decision making and predictive modelling. Some of the most common data mining techniques I discovered are: Classification - A technique used to categorise data into predefined classes based on the data's features. Association Rule - Discovering interesting relationships or patterns among a set of items in transactional or market basket data. It helps identify frequently co-occurring items and generates rules to show associations between items, such as if  X, then Y. Anomaly detection - Aims to identify unusual data instances that deviate significantly from the expected patterns. Some use cases may be detecting fraudulent actions or network intrusions. Clustering - To group similar data inst...

 Types of Problems Suited to Big Data Analysis Today, we covered the problems that arise with analysis from Big Data. We discussed why Big Data might be less predictive and powerful than first thought. For example, Big Data that has been collected over the past 5 years sounds like it could be trustworthy enough to form strong predictions from; however, the longer the Big Data collection has gone on, the better. If data has been collected for 20 years, it will hold much more accuracy than data collected over 5 years. This is because it can pick up trends that have appeared, then disappeared, then perhaps reappeared.  The issue that follows on from that is if you want to create accurate predictions, you have to find data that has been collected at a wide scale for decades, which is still rare in todays world but also expensive, especially if you want to start collecting it today. You would require large expensive data collection methods, long term, backed up storage facilities a...

Strategies for Limiting the Negative Effects of Big Data As previously discussed, there are negative effects of Big Data in modern society. There are strategies that are employed to limit these effects. Today, we discussed the Data Protection Act 2018 as one of the strategies. It incorporates the European Union’s General Data Protection Regulations (GDPR), which prohibit processing personal data unless it is expressly allowed by law or consent is given by the subject. It helps to ensure companies protect the data they collect from external forces. We also read an extract from the National Union of Journalists which described another strategy being the emphasis on free press and investigative journalism. This is a right that allows individuals to express, publish and share information and ideas without fear of censorship or government interference. I believe that this is just as important as explicit legislation, as it too holds companies accountable.

Implications of Big Data in Society Previously, I covered Big Data in Science, today, we looked at how Big Data is utilised in Society. Big Data has been incorporated in modern societies for years, and it is only going to play a larger role as its uses are better understood and its technology develops further. Today, Big Data is used in society across most departments, including water, energy, communication, housing and mobility to improve people’s quality of life. In policing, for example, it has been used to predict and identify where crimes are most likely to occur. This means the limited resources can be utilised more efficiently. It can also be used to identify the risks associated with particular individuals, such as those at risk of reoffending or being the victims of crimes. The police may also grasp a better potential of data collected through surveillance, such as CCTV or automatic number plate recognition. Finally, Big Data collected from social media allows police to ...

Implications of Big Data for Individuals   Firstly, it can be argued that Big Data leaves societies vulnerable and breaches civil liberties, especially in authoritarian governments that may utilise Big Data sets for immoral reasons, such as against political opponents. Big Data that has been collected by companies can be analysed by security services. These could include very personal data such as NHS health records. We watched a really good video which simplified the Investigatory Powers Act 2016 and it explained what the act would allow authorities to carry out. Regulation of Investigatory Powers Legislation related to the implications of Big Data for Individuals includes the Investigatory Powers Act 2016, which allows: ·         Security services to legally bug computers and phones with a warrant. ·         Security services to analyse bulk collections of communications data. ·...

 Limitations of Predictive Analysis   Predictive analysis is the use of data, algorithms and machine learning techniques to predict future outcomes from data. The largest inaccuracy of Big Data is the application of large data sets on individuals. The reason predictive analysis is flawed is that its not a guaranteed outcome. Despite each prediction being more accurate, the larger the data set is, it is still not the certain outcome. For example, predictive analytics for 2020 would have likely been inaccurate as COVID-19 would have changed the outcomes for supply chains, prices, global health, economies, and public opinion. I found a website that offered a comparison between the predictions and reality. it said that in 2020, global GDP was forecast to have 2.9% growth, however, it actually decreased by 3.4%. This shows that no matter how much data you have, you can never be completely sure of future outcomes. https://www.statista.com/statistics/1102889/covid-19-for...

Technological Requirements of Big Data I found an article explaining the requirements of Big Data to be collected at an exponential rate in today's world.  https://www.linkedin.com/pulse/technological-requirements-big-data-mustafa-sultani?trk=pulse-article_more-articles_related-content-card The technological requirements of Big Data are: Storage  Processing Data Integration  Storage The falling cost and rising capacity of digital storage have been made possible by the cloud. This allows for huge amounts of data to be stored on server farms. These cloud services should limit the chance of data being lost due to disk or system failure. Processing Improved computer power has made it possible for more powerful software to analyse huge data sets. This powerful software, such as Hadoop and Spark, can be used for analysing, processing and extracting data from an extremely complex data set. Data Integration Big Data integration is now a practice in all larg...

 Future Applications of Big Data In previous posts, I have spoken about how Big Data was used in the past, and in the modern day. This post will be about how Big Data may be used in the future. At the moment, the Big Data analytics market is reaching $350 billion, and by 2029, it is expected to grow to $655 billion. This is due to the predictions of the future applications of Big Data. For example, before I spoke about the use of Big Data in society including retail applications. Today, Big Data is used mainly for personalised marketing, inventory optimisation, and trend predictions. This approach enhances customer experience and operational efficiency. As more retail businesses around the globe become more data focused, and millions of customers are purchasing products and services from those companies, Big Data will only become more essential for the profit and growth of the retail industry. Big Data can help analyse why these customers are buying these products and who is buying...

 Big Data in Society Big Data has been incorporated in modern societies for years, and it is only going to play a larger role as its uses are better understood and its technology develops further. Today, Big Data is used in society across most departments, including water, energy, communication, housing and mobility to improve p eople’s quality of life. This video explained how Big Data has transformed policing.  60-Second Lecture: "Big Data and Policing"   It explained how Big Data has been used for crime statistics, calls for help, accidents, complaints and use of force. It has been used to predict and identify where crimes are most likely to occur.  When reading about Big Data, S age Journals suggested that Big Data technologies make probabilistic inferences about the place and time that a crime is most likely to occur.  This means the limited resources can be utilised more efficiently. It can also be used to identify the risks associated with particular in...

  Contemporary Applications of Big Data in Business Today, we looked at how Big Data can be applied to Business. It increases opportunities for machine learning, predictive analytics, data mining, etc. This translates to businesses better understanding customers, identifying operational issues, detecting fraudulent transactions and managing supply chains.  The overall results of utilising Big Data in business environments involve more effective marketing and advertising campaigns, improved business processes, increased revenue, reduced costs, and strategic planning. A good example that I found of Big Data being used in business was in a retail environment.  The retail sector generates enormous volumes of data, and its uses span across: Behaviour analysis - To analyse customer behaviour Inventory Management - To predict their demand for stock. Pricing adjustments - To understand competitors' pricing. Supply Chain Management - To reduce wait times. Market Trend Analysis...

Characteristics of Big Data Analysis   Today we learnt that Big Data analysis has two perspectives: Decision-oriented – more similar to traditional business intelligence. Look at selective subsets and representations of larger data sources and try to apply the results to the process of making business decisions. It is to augment decision-making.    Action-oriented – used for rapid response when a pattern emerges, or specific kinds of data are detected and action is required. Big data is leveraged through analysis to drive proactive or reactive behaviour changes, offering great potential for early adopters. We discussed how different so ftware, such as Hadoop and Spark, is adapted for various preferences and how they may be used as open-source frameworks for big data architectures to prepare, process, manage and analyse big data sets. We watched a video explaining what Hadoop is and when to use it.  What i...