Parquet & DuckDB for Big Data Work

You know SQL Server. You write queries every day. Now meet two small tools that handle large analytical workloads without a heavy database server.

1. The simple idea

Parquet is a file format. It stores data in columns instead of rows and compresses it well.

DuckDB is an embedded analytical database. It runs inside your application (or as a simple process) and speaks SQL.

Together they work like this:

CSV / Database Files / Logs
           ↓
        Parquet
 (columnar + compressed)
           ↓
         DuckDB
     Analytical SQL
           ↓
  Reports / Dashboards

You write normal SQL. DuckDB reads the Parquet files directly.

2. Why Parquet is powerful

Traditional files (and many databases) store data row by row:

CSV / Row-oriented

Row → Timestamp | Service | Level | Message | UserId
Row → Timestamp | Service | Level | Message | UserId
Row → Timestamp | Service | Level | Message | UserId

Parquet stores data column by column:

Parquet

Timestamp  → [2026-09-01T10:01, 2026-09-01T10:02, ...]
Service    → [OrderApi, PaymentApi, OrderApi, ...]
Level      → [Error, Info, Error, ...]
Message    → [Timeout, OK, NullRef, ...]
UserId     → [42, 17, 42, ...]

When you run an analytical query such as “count errors by service”, the engine mostly reads the Service and Level columns. It skips the rest. This reduces I/O.

Parquet also uses efficient encoding and compression. For many analytical workloads this means smaller files and less data moved from disk or object storage.

3. DuckDB + Parquet

DuckDB can query Parquet files directly. No need to load everything into a traditional table first.

Example - server error summary

SELECT
    Service,
    COUNT(*) AS ErrorCount,
    COUNT(DISTINCT Message) AS DistinctErrors
FROM 'server_errors/**/*.parquet'
WHERE Level = 'Error'
GROUP BY Service
ORDER BY ErrorCount DESC;

Example - request / response sampling

SELECT
    Service,
    COUNT(*) AS RequestCount,
    AVG(ResponseTimeMs) AS AvgResponseTime,
    approx_quantile(ResponseTimeMs, 0.95) AS P95ResponseTime
FROM 'request_logs/**/*.parquet'
WHERE Timestamp >= '2026-09-01'
GROUP BY Service;

Example - audit log activity

SELECT
    Action,
    COUNT(*) AS EventCount,
    COUNT(DISTINCT UserId) AS UniqueUsers
FROM 'audit_logs.parquet'
GROUP BY Action
ORDER BY EventCount DESC;

You can point DuckDB at one file or at a whole folder of partitioned Parquet files using a glob pattern.

4. Why this works well for large data

Many GB / TB of historical data
              ↓
         Parquet files
              ↓
            DuckDB
              ↓
         SQL analytics
              ↓
        Small result set
              ↓
         Web dashboard

DuckDB reads only the columns and row groups it needs. Your .NET application receives a small result set. It does not have to load the entire dataset into memory.

This pattern works well for historical logs, audit trails, and sampled request data that live as files (local disk, Azure Blob, S3, etc.).

5. Real-world use cases

• Microservice and application log analytics
• Server error and exception tracking
• Request / response time sampling and latency reports
• Audit log and security event analysis
• Sales and transaction reporting
• Historical permit or inspection analytics
• Data exports and simple ETL
• Local data exploration by developers
• Batch reporting jobs
• Analytics over files stored in object storage

6. Parquet vs CSV

CSV is fine for small exports and human inspection. Parquet is built for repeated analytical queries over larger volumes.

7. Key takeaway

Parquet is the storage format. DuckDB is the analytical engine.

Together they let you run familiar SQL over large collections of files without always standing up a traditional database server. You keep historical data cheaply in compressed columnar files and query only what you need when you need it.

Practical .NET project idea

Build a small ASP.NET Core MVC analytics dashboard that:

• Points DuckDB at a folder (or Azure Blob) of Parquet files containing audit logs, server errors, and request samples
• Runs a few summary SQL queries
• Shows simple charts or tables for error counts by service, average response times, and recent audit actions

This gives you a concrete way to try Parquet + DuckDB inside a familiar .NET web application.