Elasticsearch & ELK Stack Guide
Elasticsearch & ELK Stack Guide
When developers first hear Elasticsearch, they often hear:
"It is a database."
Then they hear:
"ELK is for logging."
Both statements are partly true, but they miss the bigger picture.
Elasticsearch is better understood as a distributed search and analytics engine that stores and queries document-oriented data. The broader Elastic platform can be used for application search, centralized logging, observability, security analytics, geospatial search, and AI/RAG workloads.
For a .NET microservices developer, this makes Elasticsearch much more interesting than simply another place to put log files.
This article starts from the basics and gradually builds a picture of where Elastic can fit into a real enterprise microservices architecture.
1. Is Elasticsearch a database?
Short answer
Yes, but with an important qualification.
Elasticsearch stores data, retrieves data, indexes data, and provides querying and aggregation capabilities. In that sense, it behaves like a database.
However, its primary design goal is search and analytics, rather than relational transactions.
A useful comparison is:
| Technology | Main purpose |
|---|---|
| SQL Server | Relational data, transactions, business systems |
| PostgreSQL | Relational data, transactions, business systems |
| MongoDB | Document-oriented application data |
| Redis | In-memory key/value storage and caching |
| Elasticsearch | Search, analytics, and document-oriented indexing |
For an enterprise application, you would usually not replace SQL Server with Elasticsearch just because Elasticsearch can store data.
A more common architecture is:
SQL Server
Source of Truth
|
| Indexing / synchronization
v
Elasticsearch
Search / Analytics
|
v
Search API
|
v
User
SQL Server remains responsible for transactional business data.
Elasticsearch provides a highly optimized representation of that data for search and analytics.
2. SQL Server vs Elasticsearch
Suppose your application contains permits.
In SQL Server you might have:
PERMIT
--------------------------------
Id
PermitNumber
Description
Status
District
CreatedDate
A simple search could be:
SELECT *
FROM Permit
WHERE Description LIKE '%bridge construction%';
That can work perfectly well for simple searches.
But enterprise search often becomes more complicated.
You may want:
- Full-text search
- Typo tolerance
- Relevance ranking
- Fuzzy matching
- Autocomplete
- Synonyms
- Faceted filtering
- Aggregations
- Geographic search
- Semantic search
- Vector search
This is where Elasticsearch becomes useful.
For example, a user could type:
highwy constrction permit
and a search engine can potentially understand that the user intended:
highway construction permit
The important point is:
SQL Server is excellent at managing transactional business data. Elasticsearch is designed to make large collections of documents extremely searchable and analyzable.
3. How Elasticsearch organizes data
A relational database typically looks like:
Database
|
+-- Table
|
+-- Row
Elasticsearch uses concepts such as:
Elasticsearch
|
+-- Index
|
+-- Document
A document is commonly represented as JSON:
{
"id": 101,
"name": "Road Construction Permit",
"description": "Highway construction permit for Wyoming",
"status": "Approved"
}
The index contains many documents that have a related structure.
Think of an index as a searchable collection of documents.
4. What is ELK?
The classic term ELK stands for:
E = Elasticsearch
L = Logstash
K = Kibana
Each component has a different job.
Application
|
v
Logstash
|
v
Elasticsearch
|
v
Kibana
Elasticsearch
Stores, indexes, searches, and analyzes the data.
Logstash
Collects, transforms, enriches, and routes data.
Kibana
Provides a user interface for searching, analyzing, visualizing, and building dashboards over Elastic data.
5. ELK vs Elastic Stack
You will also hear the term Elastic Stack.
Historically, ELK was the most common shorthand:
Elasticsearch
Logstash
Kibana
The broader Elastic ecosystem now includes additional components and capabilities, such as:
Elasticsearch
Kibana
Logstash
Elastic Agent
APM
Elastic Observability
Elastic Security
So:
ELK
|
+-- Elasticsearch
+-- Logstash
+-- Kibana
is better viewed as part of the larger:
Elastic Stack
6. The biggest misconception: ELK is only for logs
Logging is one of the most common Elastic use cases.
But it is only one part of the story.
A better mental model is:
ELASTIC
|
+--------------+--------------+
| | |
v v v
SEARCH OBSERVABILITY SECURITY
| | |
| | |
Full-text Logs SIEM
Semantic Metrics Threats
Vector Traces Alerts
Geo APM Investigation
Analytics Monitoring
This is why Elasticsearch can become the foundation for an entire product rather than merely a logging destination.
7. Where can ELK fit into a .NET microservices project?
Suppose your architecture looks like this:
Browser
|
v
+-----------+
| Ocelot |
| Gateway |
+-----+-----+
|
+-----------------+------------------+
| | |
v v v
Auth API ROW API CMS API
| | |
+-----------------+------------------+
|
Other services
|
+-------------+-------------+
| |
v v
Service Bus Redis
|
v
Workers
Elastic can sit beside this architecture:
Browser
|
v
Ocelot
|
+-----------------+------------------+
| | |
v v v
Auth API ROW API CMS API
| | |
+-----------------+------------------+
|
v
Application Events
|
v
Elastic / OpenTelemetry
|
+--------------+--------------+
| | |
v v v
Logs Metrics Traces
| | |
+--------------+--------------+
|
v
Kibana
You can therefore use Elastic for:
- Centralized logging
- Distributed tracing
- Application performance monitoring
- Application search
- Business analytics
- Security analytics
- Geospatial search
- AI/RAG retrieval
8. Use case #1: Centralized logging
Imagine you have 10 or 20 microservices.
Without centralized logging:
Auth API logs
ROW API logs
CMS API logs
Notification API logs
Gateway logs
Worker logs
...
When something breaks, you may have to inspect many different places.
With centralized logging:
Auth API
ROW API
CMS API
Gateway
Workers
|
v
Centralized logging
|
v
Elasticsearch
|
v
Kibana
Now you can search all services from one place.
For example:
service: rowapi
level: error
or:
recordId: 12345
9. Structured logging
A major improvement is to avoid treating logs as plain text.
Instead of:
_logger.LogInformation(
$"Record {recordId} was approved by {userId}");
prefer structured logging:
_logger.LogInformation(
"Record approval requested. RecordId={RecordId}, UserId={UserId}",
recordId,
userId);
This lets your logging pipeline preserve fields such as:
RecordId = 12345
UserId = 789
Instead of storing only one long string.
A document might eventually look like:
{
"service": "rowapi",
"level": "Information",
"message": "Record approval requested",
"recordId": 12345,
"userId": 789,
"timestamp": "2026-09-16T14:30:00Z"
}
Now Kibana can search and aggregate those fields.
10. Use case #2: Correlation IDs
This is one of the most valuable ideas for microservices.
Consider this request:
Browser
|
v
Ocelot
|
v
ROW API
|
v
Notification API
|
v
Service Bus
|
v
Notification Worker
The original request might have:
CorrelationId = abc-123
Every service includes that ID in its logs.
Then Kibana can search:
CorrelationId: "abc-123"
You might see:
14:29:58 Gateway
POST /row/record/approve
14:29:59 ROW API
Approval request received
14:30:00 ROW API
Review validation started
14:30:01 ROW API
Review validation failed
14:30:01 ROW API
HTTP 400 returned
Instead of searching service by service, you can follow the request through the distributed system.
This is one of the biggest practical advantages of centralized observability.
11. Use case #3: Observability
Modern observability is broader than logging.
A useful model is:
Observability
|
+-- Logs
|
+-- Metrics
|
+-- Traces
Logs
Tell you what happened.
Payment failed.
Metrics
Tell you how much/how often.
HTTP 500 rate = 2.4%
Traces
Tell you where time was spent across a distributed request.
For example:
Request abc-123
|
+-- Gateway 12 ms
|
+-- ROW API 850 ms
| |
| +-- EF Query 720 ms
| +-- Business 100 ms
|
+-- Response 20 ms
Now you can see that the database query is responsible for most of the latency.
This is much more useful than simply knowing that the request was slow.
12. Serilog + Elasticsearch
For a .NET application, one possible approach is:
.NET API
|
v
Serilog
|
v
Elasticsearch
|
v
Kibana
Your application can produce structured logs.
Conceptually:
_logger.LogInformation(
"Permit created. PermitId={PermitId}, UserId={UserId}",
permitId,
userId);
The exact ingestion architecture can vary.
You do not necessarily need Logstash for every .NET application.
For example, depending on requirements:
.NET
|
+-- Serilog
|
+-- OpenTelemetry
|
+-- Elastic Agent / Collector
|
v
Elasticsearch
|
v
Kibana
The correct pipeline depends on the type and volume of telemetry you are collecting.
13. Use case #4: Application search
This is where Elasticsearch starts becoming an actual application feature, not just infrastructure.
Imagine a permitting system containing:
Permits
Agreements
Licenses
Documents
Users
Requirements
Policies
A user can search:
bridge construction
and receive relevant results across multiple document types.
The architecture could be:
SQL Server
|
| indexing
v
Elasticsearch
|
v
Search API
|
v
Web application
SQL Server remains the source of truth.
Elasticsearch contains a search-optimized representation.
14. Full-text search
A search engine can provide capabilities such as:
"bridge construction"
rather than relying only on:
LIKE '%bridge construction%'
You can add:
- Tokenization
- Stemming
- Fuzzy matching
- Relevance scoring
- Synonyms
- Autocomplete
- Field boosting
- Filtering
This becomes particularly useful when users don't know exactly what the database field contains.
15. Use case #5: E-commerce search
A classic large-scale use case is product search.
Imagine:
100 million products
A user searches:
nike black running shoes under 150
The system may combine:
Text
+
Brand
+
Color
+
Category
+
Price
+
Rating
+
Availability
+
Relevance
The search experience might look like:
Search: nike black running shoes under $150
--------------------------------------------
Nike Running Shoe
$129
Nike Trail Shoe
$139
Nike Air Zoom
$149
This is a very different problem from simply fetching a row from SQL Server.
16. Use case #6: Geospatial search
Elasticsearch can also be used for geographic queries.
Suppose your application stores:
Permit
Latitude
Longitude
Route
Milepost
District
County
Geometry
A user could ask:
Show permits within 5 miles of this location.
Conceptually:
Map
|
v
Search API
|
v
Elasticsearch
/ \
/ \
Geo-distance Geo-shape
\ /
\ /
Results
This can be particularly useful for transportation, utilities, field operations, asset management, and permitting systems.
17. Use case #7: Business analytics
Elasticsearch is also useful for aggregating large volumes of events.
Suppose your system produces:
1 million events/day
You might want to answer:
How many permits were created today?
Which district created the most permits?
What percentage are approved?
Which API has the most errors?
What is the average approval time?
Which hour has the highest traffic?
Kibana can turn these queries into dashboards.
For example:
+------------------------------------------------+
| SYSTEM DASHBOARD |
+-------------+-------------+--------------------+
| 12,542 | 1,245 | 98.7% |
| Permits | Approved | Success Rate |
+-------------+-------------+--------------------+
| |
| Requests per hour |
| /\ |
| ___/ \____ |
| |
+----------------------+-------------------------+
| District | Status |
| District 1 | Approved |
| District 2 | Review |
| District 3 | Started |
+----------------------+-------------------------+
18. Use case #8: Security analytics / SIEM
This is another major area.
A security platform can ingest events from:
Windows
Linux
Firewalls
Cloud platforms
Applications
Authentication systems
Endpoints
Network devices
Then security teams can search for patterns.
For example:
Login failed
|
+-- 5 attempts
|
+-- same IP
|
+-- multiple accounts
|
v
Potential security event
Elastic Security provides capabilities for security analytics, threat detection, investigation, and response.
A large organization can therefore use Elastic as part of a security operations platform rather than simply using it for application logs.
19. Use case #9: AI and RAG
This is one of the most interesting modern use cases.
Suppose you have:
Policies
Manuals
Requirements
Training documents
FAQs
Tickets
Technical documentation
You want an AI assistant that can answer questions using those documents.
The architecture could be:
User
|
v
AI Application
|
v
Elasticsearch
/ | \
/ | \
Keyword Semantic Vector
Search Search Search
\ | /
\ | /
Relevant Docs
|
v
LLM
|
v
Answer
This is a RAG-style architecture.
The important point is that the LLM does not need to contain every piece of your organization's knowledge.
Elasticsearch can act as the retrieval layer that finds relevant information.
20. Vector search
Traditional search might look for matching words.
For example:
User:
How do I approve an agreement?
A keyword search looks for terms such as:
approve
agreement
Semantic/vector search can retrieve content based more on the meaning of the request.
Conceptually:
Text
|
v
Embedding
|
v
Vector
|
v
Vector index
|
v
Similar documents
This is one of the reasons Elasticsearch is increasingly relevant to AI applications.
21. A complete AI knowledge assistant
You could build an enterprise assistant like:
Enterprise Knowledge Assistant
User
|
| "What happens when an agreement is submitted?"
v
AI API
|
v
Elasticsearch
|
+-- Search policies
+-- Search requirements
+-- Search documentation
+-- Semantic retrieval
|
v
Relevant documents
|
v
LLM
|
v
Answer + source references
This can become a complete application rather than a small demo.
22. Use case #10: Build a Splunk-like platform
Another large project idea is to build your own internal log-analysis platform.
Conceptually:
Servers
Applications
Containers
Azure
Windows
Linux
Microservices
|
v
Elastic Agent / Logstash
|
v
Elasticsearch
|
v
Kibana
|
+-- Search
+-- Dashboards
+-- Alerts
+-- Analytics
+-- Investigation
The result is essentially an internal observability/log analytics platform.
This is the type of architecture that can support very large environments.
23. Are there real large-scale Elastic projects?
Yes.
Elastic publishes customer examples involving large deployments.
Examples include organizations using Elastic for:
- Security analytics
- Observability
- Search
- Recommendations
- Large-scale event analysis
For example, Elastic's customer material describes Siemens using Elastic Security to correlate billions of security events per day across a very large endpoint environment.
The important lesson is not the exact customer architecture.
The lesson is:
Elasticsearch is capable of being part of the core infrastructure of very large systems.
It is not merely a developer logging utility.
24. A complete project for a .NET developer
If you want one project that teaches most of these concepts, build an:
Enterprise Microservices Intelligence Platform
The architecture could look like:
+----------------+
| Web UI |
+-------+--------+
|
+----------------+----------------+
| | |
v v v
Search Observability Security
| | |
+----------------+----------------+
|
+-------v--------+
| Elasticsearch |
+-------+--------+
|
+-------------------+-------------------+
| | |
v v v
Logs Metrics Traces
^ ^ ^
| | |
+-----+-------------------+-------------------+-----+
| .NET Microservices |
| |
| Gateway | Auth | ROW | CMS | Notification | ... |
+--------------------------------------------------+
Then add GIS:
Elasticsearch
|
+----------+----------+
| |
Search Geo Search
| |
v v
Documents Map UI
Then add AI:
Elasticsearch
|
Relevant content
|
v
LLM
|
v
AI Assistant
Now you have one platform demonstrating multiple Elastic capabilities.
25. Suggested modules for the project
Module 1 - Centralized Logging
Build:
.NET APIs
|
v
Structured logging
|
v
Elasticsearch
|
v
Kibana
Features:
- Log levels
- Structured fields
- Exception logging
- Correlation ID
- Service name
- Environment
- Request information
Module 2 - Microservice Observability
Add:
Logs
+
Metrics
+
Traces
Track:
Request duration
HTTP status
Database duration
External API duration
Exception rate
Service dependencies
Module 3 - Enterprise Search
Index:
Permits
Agreements
Documents
Policies
Users
Implement:
Full-text search
Fuzzy search
Filters
Sorting
Aggregations
Autocomplete
Module 4 - GIS Search
Add:
Latitude
Longitude
Route
Milepost
Geometry
Implement:
Nearby search
Radius search
Bounding-box search
Geo filtering
Module 5 - Security Analytics
Index:
Login events
Authentication failures
Audit events
API security events
Create Kibana views for:
Failed logins
IP activity
User activity
Authentication patterns
Security alerts
Module 6 - AI/RAG
Index:
PDF
Markdown
HTML
Policies
Technical documentation
FAQs
Implement:
Keyword retrieval
Semantic retrieval
Vector retrieval
Hybrid search
LLM answer generation
Source references
26. What this teaches you
A single project like this teaches:
Elasticsearch
|
+-- Indices
+-- Documents
+-- Mappings
+-- Queries
+-- Aggregations
+-- Full-text search
+-- Fuzzy search
+-- Geo search
+-- Vector search
|
+-- Kibana
|
+-- Logstash / Agent
|
+-- Structured logging
|
+-- OpenTelemetry
|
+-- Distributed tracing
|
+-- Security analytics
|
+-- RAG
It is considerably more educational than building:
.NET API
|
v
Log
|
v
Elasticsearch
and stopping there.
27. Recommended learning path
Do not start with everything at once.
Build it incrementally.
Phase 1 - Elasticsearch fundamentals
Learn:
Index
Document
Mapping
Field
Query
Aggregation
Shard
Replica
Build a small .NET search API.
Phase 2 - .NET + Elasticsearch
Create:
ASP.NET Core API
|
v
Elasticsearch
Implement:
Create document
Get document
Search
Filter
Sort
Aggregate
Phase 3 - Kibana
Learn:
Discover
Dashboards
Visualizations
Queries
Filters
Alerts
Phase 4 - Centralized logging
Add:
.NET
|
Serilog / structured logging
|
Elasticsearch
|
Kibana
Phase 5 - Microservices observability
Add:
Correlation IDs
OpenTelemetry
Traces
Metrics
Service dependencies
Phase 6 - Advanced search
Add:
Fuzzy search
Autocomplete
Synonyms
Relevance
Aggregations
Phase 7 - Geo search
Add:
Coordinates
Distance queries
Geo shapes
Map visualization
Phase 8 - Security
Add:
Authentication events
Audit logs
Security analytics
Detection rules
Alerts
Phase 9 - AI
Add:
Embeddings
Vector search
Semantic search
Hybrid search
Phase 10 - RAG
Finally:
Documents
|
v
Elasticsearch
|
v
Retrieval
|
v
LLM
|
v
Enterprise AI Assistant
28. One important architectural rule
Do not fall into this trap:
SQL Server
|
X
Replace everything with Elasticsearch
Instead:
SQL Server
Source of Truth
|
Data / Events
|
v
Elasticsearch
|
+------------+-------------+
| | |
Search Analytics Observability
Each technology has a job.
For example:
SQL Server
-> Transactions and relational business data
Redis
-> Cache / fast temporary state
Service Bus
-> Messaging / asynchronous communication
Elasticsearch
-> Search / analytics / telemetry
Kibana
-> Explore / visualize / investigate
OpenTelemetry
-> Collect telemetry
LLM
-> Generate AI responses
This is a much healthier microservices architecture than trying to make one technology solve every problem.
29. The biggest takeaway
If someone asks:
"What is Elasticsearch?"
A basic answer is:
"It is a search engine/database for JSON documents."
A better answer is:
"Elasticsearch is a distributed search and analytics engine that can power application search, observability, security analytics, geospatial queries, and AI retrieval."
And if someone asks:
"What is ELK?"
The traditional answer is:
Elasticsearch
+
Logstash
+
Kibana
But the bigger picture is the Elastic ecosystem:
ELASTIC
|
+---------------+---------------+
| | |
v v v
SEARCH OBSERVABILITY SECURITY
| | |
| +----+----+ |
| | | | |
| Logs Metrics Traces SIEM
| | |
+---------------+---------------+
|
v
Elasticsearch
|
v
Kibana
|
v
Human / Application
That is why Elastic is worth learning as a platform, not merely as a logging tool.
30. Final project idea
For a .NET developer working with microservices, a strong end-to-end learning project is:
Enterprise Microservices Intelligence Platform
Start with:
.NET 8/10
Ocelot
SQL Server
Redis
Elasticsearch
Kibana
Then add:
Serilog
OpenTelemetry
Service Bus
Geo search
Security analytics
Vector search
RAG
A final architecture might look like:
Web Application
|
v
Ocelot Gateway
|
+----------------+----------------+
| | |
v v v
Auth API ROW API CMS API
| | |
+----------------+----------------+
|
+--------+--------+
| |
v v
SQL Server Redis
|
|
Business Events
|
v
Elasticsearch
|
+--------------+--------------+
| | |
v v v
Search Observability Security
| | |
| Logs/Metrics/ SIEM
| Traces |
+--------------+--------------+
|
v
Kibana
|
+--------+--------+
| |
v v
Dashboards Analysis
+
|
v
Vector / Semantic
Retrieval
|
v
LLM
|
v
AI Assistant
You can build the entire learning environment locally with Docker Desktop before introducing Azure.
That progression takes you from "What is Elasticsearch?" to actually understanding why companies deploy Elastic as a major part of their infrastructure.
Official references
- Elasticsearch: https://www.elastic.co/elasticsearch
- Elastic Stack: https://www.elastic.co/docs/get-started/the-stack
- Elastic Solutions: https://www.elastic.co/docs/solutions
- Elastic Observability: https://www.elastic.co/docs/solutions/observability
- Elastic Security: https://www.elastic.co/docs/solutions/security
- Elastic customer stories: https://www.elastic.co/customers
Enjoyed this article? Share it with your network!