What Is Big Data? Definition, the Vs, Examples and Careers

Big data defined

There is no single official definition, but a widely cited one comes from the U.S. National Institute of Standards and Technology (NIST). In its Big Data Interoperability Framework, NIST says big data “consists of extensive datasets” whose main characteristics are volume, variety, velocity and/or variability, and that “require a scalable architecture for efficient storage, manipulation, and analysis.”

Notice what the definition does not say. It gives no minimum size. Data becomes “big” when your current tools can no longer handle it well, not when it passes a number of terabytes.

Where the term’s core idea came from

The three-part idea predates the buzzword. Doug Laney, then an analyst at Meta Group (now part of Gartner), described the “3-dimensional data challenge” of rising data volume, velocity and variety in conference talks in the late 1990s. He published it in a research note dated 6 February 2001, titled 3-D Data Management: Controlling Data Volume, Velocity and Variety. Laney tells that history in his own blog post.

The Vs of big data

People describe big data with a list of words that start with V. Three are the core. NIST adds a fourth.

The examples are our illustrations, not NIST’s.

What about veracity and value?

You will often see five or even seven Vs, adding veracity (is the data trustworthy?) and value (is it worth the cost?). They matter. But Laney, who coined the original three, argues they are important concepts that should not be treated as part of what defines big data. Quality and value apply to every dataset, big or small.

A quick test: is my data “big”?

If every answer is no, a normal database and a spreadsheet may be enough. That is a good outcome, because big data systems cost more to build and run.

Structured, semi-structured and unstructured data

Variety shows up in three broad forms.

Much of the growth in big data comes from the second and third types, which traditional databases handle poorly. Descriptive information about data, called metadata, helps make all three findable and usable. See why metadata matters in a data-driven world.

Where big data comes from

• Transactions: purchases, payments, bookings and orders.
• Web and app activity: clicks, searches, page views and session logs.
• Sensors and devices: machines, vehicles, wearables and other connected equipment.
• Social and text data: posts, reviews, support tickets and emails.
• Media: images, video and audio.
• Public and shared datasets: government statistics, research data and partner feeds.

Much of this is personal data, which raises legal and ethical questions. Read six ways companies collect your data to see how it is gathered.

How big data is stored and processed

The key idea in NIST’s wording is a scalable architecture. Instead of one large computer, big data systems spread data and work across many machines. NIST calls this the big data paradigm: data systems distributed across horizontally coupled, independent resources, to reach the scalability needed to process extensive datasets.

In practice, a typical setup has four layers:

1. Collection: data is gathered from source systems, often as a continuous stream.
2. Storage: raw data goes into a data lake, which holds data in its original form, and cleaned, organized data goes into a data warehouse.
3. Processing: distributed frameworks such as Apache Hadoop and Apache Spark split the work across many machines. Many organizations run these on cloud platforms instead of their own hardware.
4. Analysis and presentation: analysts and data scientists use queries, statistics, machine learning and dashboards to turn the data into decisions.

Tool names change often, so learn the layers first. They stay the same when the products change.

Big data vs. business intelligence

The two terms overlap but are not the same thing.

Many organizations use both: big data systems collect and prepare the data, and BI tools present the results to decision makers. Our guide to the components of business intelligence shows how the pieces fit.

Common uses

These are typical patterns, not results you can expect from any single project.

Challenges

• Data quality: more data does not mean better data. Errors and duplicates scale up with the volume.
• Privacy and security: personal data must be protected, and the rules differ by country and industry.
• Cost and complexity: storage, computing and specialist staff add up.
• Skills: people who can build and interpret these systems are in demand.
• Governance: without clear ownership and documentation, a data lake becomes a swamp that nobody trusts.
• Unclear goals: collecting data first and finding a question later rarely pays off. Start with the decision you want to improve.

Careers and pay

The U.S. Bureau of Labor Statistics (BLS) does not use the term “big data” as an occupation. The closest match is data scientists, whose listed duties include collecting, cleaning and analyzing data, building models including machine learning models, and presenting findings to stakeholders.

These figures describe one occupation. Other roles that work with big data, such as data engineers and analysts, are counted in different categories. Pay depends on industry, location and experience, and a course does not guarantee a job.

If your interest is the business side, our business intelligence certificate guide is a good next step.

Learn big data online

You can test the field cheaply before choosing a degree or certificate. Start with the basics: databases and SQL, statistics, and one programming language such as Python. Then explore distributed data tools.

• Browse big data courses on Coursera
• Browse big data courses on Udemy

Disclosure: some links on this page are affiliate links. If you buy through them, we may earn a commission at no extra cost to you. VillanovaU is an independent guide and is not affiliated with NIST, Gartner, any university or the platforms listed. Course quality varies, so read each syllabus first.

Frequently asked questions

What is big data in simple terms?

It is data that is too large, too fast or too varied for ordinary tools to store and analyze well, so it needs systems that spread the work across many computers.

What are the 3 Vs of big data?

Volume (how much data), velocity (how fast it arrives) and variety (how many types and sources). They were described by Doug Laney in a research note dated 6 February 2001.

What are the 4 Vs, 5 Vs and 7 Vs?

NIST adds variability to the three core Vs. Other lists add veracity and value, among others. Laney says those extras matter but should not be treated as defining big data.

How big is big data?

There is no fixed size. Data counts as big when your current tools cannot handle it efficiently.

What is the difference between big data and data science?

Big data describes the data. NIST describes data science as the methodology for synthesizing useful knowledge directly from data, through discovery or hypothesis testing. Data scientists often work with big data, but they also work with small datasets.

Is big data the same as business intelligence?

No. Business intelligence mostly reports on what has happened using structured data. Big data analytics handles larger, faster or more varied data, often to find patterns and make predictions. Many organizations use both.

Do I need a degree to work with big data?

BLS lists a bachelor’s degree as the typical entry requirement for data scientists, and some employers prefer a graduate degree. Other roles vary by employer.