Big Brain - Big Data
Sometimes data is so complex that it becomes difficult to manage using conventional database tools.
This might be because there is so much of it (for example web analytics data). It might also be because the data is not structured in a way that works nicely with a database (for example a directory of images or documents).
A server powerful enough to handle large data volumes may be expensive, especially when things like backups and high availability are considered. It is often better to use an array of lower cost servers instead, storing data on multiple machines so if any one fails others can take over. Big Brain Intelligence has used Cassandra and HDFS clusters for this.
To gain information from this data, it needs to be converted into a form that can be managed using conventional tools. You might convert a large directory of documents into a small database table containing a summary that is easier to analyse. This may need to be updated in real time.
Alternatively, you may have a task you want to perform on your data that takes a long time. You may split this across multiple servers to run more quickly and combine the results using techniques such as MapReduce.
Big Brain Intelligence can work with you to understand what data you have or can collect and consider the decisions you might make to drive your business. We can then create the technology to collect the data and convert it to information you can use to make the decisions.
Example - RFID Devices
RFID devices generate vast amounts of data when tags pass within the range of a reader. Big Brain Intelligence has developed a system to add intelligence to a reader (using a C on a Raspberry Pi) and efficiently record and process the summarised data.
Example - Online Training
Online training and subsequent discussions surrounding the training generate large amounts of data. Big Brain Intelligence has consulted with an organisation providing this to develop a hybrid approach, storing relationally structured data in SQL and data with high volumes in Azure Table and Blob storage.