According to analyst firm Gartner,
"The average organization loses $8.2 million annually through poor Data Quality".
And yet, according to the Experian Data Quality report, "99% of organizations have a data
quality strategy in place".
This is disturbing in that these Data Quality practices
are not finding the bad data that exists in their data. This is a problem that needs to be solved.
Performance Testing
Performance Testing for Big Data includes two main actions:
Data ingestion and Throughout: In this stage, the tester verifies
how the fast system can consume data from various data source. Testing involves
identifying different message that the queue can process in a given time frame. It
also includes how quickly data can be inserted into underlying data store for
example insertion rate into a Mongo and Cassandra database.
Data Processing: It involves verifying the speed with which the
queries or map reduce jobs are executed. It also includes testing the data
processing in isolation when the underlying data store is populated within the data
sets. For example running Map Reduce jobs on the underlying HDFS
Sub-Component Performance: These systems are made up of multiple
components, and it is essential to test each of these components in isolation. For
example, how quickly message is indexed and consumed, map reduce jobs, query
performance, search, etc.
Performance Testing Approach
Performance testing for big data application involves testing of huge volumes of structured and unstructured data and it requires a specific testing approach to test such massive data.
Performance Testing is executed in this order
Parameters for Performance Testing
Various parameters to be verified for performance testing are
Test Environment needs depend on the type of application you are testing. For Big data
testing, test environment should encompass:
| Big Data Cluster | Big Data Tools |
|---|---|
|
|