HBase: Hadoop's Built-In Database Engine

Your Spark jobs churn through petabytes of data on HDFS every night — but the moment product asks for a dashboard that shows any user's last 50 actions in under 100ms, batch processing falls flat. HBase is the answer to that exact problem: a distributed, strongly consistent store built directly on top of HDFS that gives you random read/write access at scale, without leaving the Hadoop ecosystem.

Where HBase Fits in the Big Data Stack

HBase was modelled after Google's Bigtable paper, published in 2006. Google needed a system to store and retrieve massive amounts of structured data — web crawl results, satellite imagery, user behaviour — with random access at any row. HBase brought that same model into the open-source Hadoop world.

You already know Cassandra from Lesson 18. Both are column-family stores. But there is a critical difference: Cassandra is purpose-built for operational, low-latency workloads from live applications. HBase is built to sit on top of HDFS and serve both batch pipelines (Spark, MapReduce) and random-access lookups from the same underlying data.

The HBase Data Model — Rows, Families, and Cells

HBase stores data in tables, but the analogy to SQL ends immediately. Every row is identified by a unique row key. Inside each row, data is grouped into column families, and within each family you can have any number of column qualifiers. The intersection of row key + column family + column qualifier + timestamp is a cell.

HBase Architecture — The Moving Parts

Understanding the architecture tells you exactly why HBase is fast and where its limits are. Three components do all the work.

Row Key Design — The Most Important Decision You'll Make

In HBase, all data is physically sorted by row key. This is both a superpower and a trap. If your row key is a timestamp or an auto-incrementing integer, all your writes land on the same region — one RegionServer gets hammered while the rest sit idle.

Hands-on — Building a Telecom CDR Store

The scenario: You are a data engineer at a telecom company. Your team needs to store call detail records — 50 million rows per day, each containing call duration, cell tower ID, and signal strength. The billing team needs to query any subscriber's full call history instantly. Spark jobs scan full days for fraud analysis. You are setting up the HBase schema and validating it end to end.

# Start HBase in standalone mode (local dev/test)
hbase start

# Open the HBase shell
hbase shell

# Create table with two column families
create 'call_records', 'meta', 'signal'

# List tables to confirm
list

HBase(main):001:0> create 'call_records', 'meta', 'signal'
Created table call_records
Took 1.4210 seconds
=> Hbase::Table - call_records

HBase(main):002:0> list
TABLE
call_records
1 row(s)
Took 0.0310 seconds

The scenario continues: You insert test CDRs to validate the schema. Row keys are encoded as phoneNumber_reverseTimestamp — the reverse timestamp ensures the most recent call for each subscriber sorts first, and spreads writes evenly across regions.

# Row key: phoneNumber_reverseTimestamp  (hotspot-safe)
put 'call_records', '447700900123_9999999987654', 'meta:duration', '142'
put 'call_records', '447700900123_9999999987654', 'meta:tower_id', 'TOWER-88'
put 'call_records', '447700900123_9999999987654', 'signal:rssi',   '-72'

# Read the row back
get 'call_records', '447700900123_9999999987654'

COLUMN                  CELL
 meta:duration          timestamp=1736512943210, value=142
 meta:tower_id          timestamp=1736512943318, value=TOWER-88
 signal:rssi            timestamp=1736512943404, value=-72
3 row(s)
Took 0.0048 seconds

The scenario continues: Production pipelines will use Python. Data engineers need to bulk-load test records from scripts running on Spark workers. You demonstrate programmatic access using the happybase library with batched writes.

import happybase

# Connect via HBase Thrift server (must be running: hbase thrift start)
connection = happybase.Connection('localhost', port=9090)
connection.open()

# Reference the table
table = connection.table('call_records')

# Bulk write using batch — collapses many puts into fewer Thrift calls
with table.batch(batch_size=100) as batch:
    for i in range(3):
        row_key = f'447700900{i:03d}_9999999987{i:03d}'.encode()
        batch.put(row_key, {
            b'meta:duration': str(120 + i * 10).encode(),
            b'meta:tower_id': f'TOWER-{80 + i}'.encode(),
            b'signal:rssi':   str(-70 - i).encode(),
        })

print("Batch written successfully")

Batch written successfully

The scenario continues: The billing team runs end-of-month jobs totalling call duration per subscriber. They need all records for one phone number across a date range. Your row key prefix design makes this a clean range scan — no secondary index required.

# Scan all records for one subscriber using prefix
# row_prefix is internally translated to a start-row / stop-row range
for key, data in table.scan(
    row_prefix=b'447700900123_',  # Only this subscriber
    columns=[b'meta:duration']    # Column projection — skip signal data entirely
):
    duration = int(data[b'meta:duration'])
    print(f"  Row: {key.decode()[:30]}...  Duration: {duration}s")

  Row: 447700900123_9999999987654...  Duration: 142s
  Row: 447700900123_9999999987000...  Duration: 198s
  Row: 447700900123_9999999986500...  Duration: 67s
Scan complete: 3 rows in 4.2ms

HBase vs Cassandra — Picking the Right Column Store

Practice Questions — You're the Engineer

Quiz — HBase in Production