Historical Development of Big Data- Before the 2000s

 Historical Development of Big Data- Before the 2000s

1800s

We explored the development of Big Data, focusing on its exponential growth and the technology required to keep up with its demands.

I read that in 1880, the technology that was accessible at the time meant that census data took seven years to compile, and thirteen years was predicted for the 1890 census, however in 1888, Herman Hollerith built an early technology that was able to analyse the data from the 1890 census much faster, resulting in the 1890 census only taking six weeks to compile all the data.

This trend of improving technology, resulting in faster and more effective data analysis, has followed humanity for the past 150 years.

1900s

We watched an interesting video from SIEMENS detailing the history of Big Data Analytics in the 1900s.

In the 20th century, we learnt that there were also false predictions of how larger data sets would be managed. The video outlined certain events across the century.

In 1944, Fremont Rider stated that at the rate libraries were growing, he estimated that large libraries such as Yale would require over 6000 catalogue staff by 2040. As we know today, companies don't require those human resources and can analyse far more data in a fraction of the time, thanks to modern technology.

In 1961, Derek Price concluded that the number of new scientific journals was growing exponentially rather than linearly, with 10 times the amount every 50 years. This reinforces the idea that the amount of data we handle is growing exponentially; however, so are the methods of its analysis.

In 1965, the Government planned the world's first data centre to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape. The technology has changed significantly since however, this was an early step to gathering extremely large data sets.

In 1996, it was assumed that digital storage had now become more cost-effective than paper. 

Comments

Post a Comment

Popular posts from this blog

Limitations of Traditional Statistics

Image
 Limitations of Traditional Statistics In my last post, I spoke about the use of traditional statistics. Now, I will cover some of the drawbacks of using traditional statistics to make predictions in the real world. Traditional data analysis has been practised with structured data, often in a relational database; however, Big Data isn’t as manageable with this structured format. Traditional statistics is about analysing and summarising data and is suited to processing smaller amounts of linear, repeatable data to draw a conclusion. The easiest environments are when the data has a stable relationship – e.g. regular exercise increases, obesity decreases. Therefore, the practicality of traditional data analysis is limited when applied to the real world. More recently, there is now the availability of new large data sets and cheaper and faster computing power. In addition, there has been a surge in machine learning research and development, which aims to automatically discover re...
Read more

Application of Big Data Techniques to a Problem

 Application of Big Data Techniques to a Problem From the previous posts, it is clear that Big Data is collected, stored and utilised in the real world at enormous scales by huge companies. Now I want to show directly how these companies use this data to address their problems in the modern world. Netflix Netflix is known for its great, vast choice of shows that users love. But how does Netflix know what each individual customer might want to watch? Netflix recommends specific titles for each user on the home page. It uses multiple algorithms for personalised rankings, searches, similarity ratings, ranking and more to narrow down the shows to display on the recommended screen for each individual customer. It looks at what titles the user has watched, searched for, and rated good or bad so the system can show them similar content. Additionally, if user A shows the same interests as user B, Netflix will show user A the user B's recommended shows. Each show is also put into some of th...
Read more

Traditional Statistics

 Traditional Statistics Today we learned about types of traditional statistics and how they're used to draw conclusions.  Statistics is the discipline that concerns the collection, organisation, analysis, interpretation, and presentation of data. It can be broken up into descriptive and inferential statistics. Descriptive Statistics Only the subjects that have been analysed have findings implied upon them. If we had the political views of 20 people, we may be interested in the voting intentions of these people or the distribution or spread of the votes, but the findings are limited to only these 20 people. Inferential Statistics If you don’t have access to the whole population you’re interested in, but only a small portion. If you want to study the political voting intentions of everybody in Scotland. From what I've learnt about the two types, I would say that descriptive is more accurate as conclusions are drawn from what the data had been directly collected from. Howev...
Read more