"We need real-time analytics" is one of the most expensive sentences in software. It conjures images of Kafka clusters, Flink pipelines, dedicated streaming teams, and infrastructure bills that make your CFO cry.

But here's what we've learned building data platforms for companies ranging from 5-person startups to logistics conglomerates operating across 100+ countries: most teams don't need real-time. They need fast-enough. And fast-enough is dramatically cheaper than real-time.

The real-time trap

When a client says "real-time," we always ask: what does that actually mean for your business?

  • "I want to see dashboard numbers update without refreshing" — that's a 30-second refresh, not real-time.
  • "I need to alert when a metric crosses a threshold" — that's near-real-time with a 1-minute check.
  • "I need sub-second fraud detection on every transaction" — OK, that's actually real-time.

    • 90% of the time, it's one of the first two. The architecture for 30-second dashboards is an order of magnitude simpler and cheaper than sub-second streaming.

    The lesson: Define your latency requirement in seconds before choosing your architecture. The number changes everything.

    The modern "fast enough" stack

    Here's what we deploy for most analytics use cases — and it handles millions of events without breaking a sweat:

    EventBridge for event routing. AWS EventBridge gives you serverless event bus with filtering rules, dead letter queues, and pay-per-event pricing. No Kafka cluster to manage, no ZooKeeper, no broker tuning. For 90% of event-driven architectures, it's the right call.

    DynamoDB Streams for change capture. When your application writes to DynamoDB, streams automatically capture every change. Lambda functions process these changes in near-real-time — typically under 1 second. No polling, no CDC infrastructure, no Debezium.

    Lambda for transformation. Serverless compute means you pay per invocation. A pipeline that processes 10,000 events per day costs pennies. A pipeline that processes 10 million costs dollars, not hundreds. And it scales automatically — no capacity planning.

    Snowflake or Redshift for analysis. Load transformed data into a cloud warehouse on a schedule — every 5 minutes, every hour, whatever your latency requirement dictates. Modern warehouses handle concurrent analytical queries without dedicated infrastructure.

    Power BI or Grafana for visualization. Dashboards that refresh on a schedule, with drill-down capabilities that let analysts explore without engineering support. Self-service analytics without a data engineering bottleneck.

    What this costs

    For a mid-size deployment processing a few million events per day with 5-minute analytics latency:

  • EventBridge: ~$5/month
  • Lambda (transformation): ~$10-30/month
  • DynamoDB Streams: included with DynamoDB
  • Snowflake (small warehouse, auto-suspend): ~$50-150/month
  • Total: under $200/month

    • Compare that to a self-managed Kafka cluster: 3 brokers minimum, ZooKeeper nodes, a Flink or Spark Streaming cluster for processing, and an engineer who understands all of it. You're looking at $2,000-5,000/month in infrastructure alone, plus the operational cost.

    The price ratio is 10-25x. For analytics that's 5 minutes behind instead of 5 seconds behind.

    When you actually need streaming

    We've built genuine streaming pipelines too. An EV charging network needs real-time telemetry — you can't wait 5 minutes to know a charger is offline. A smart energy platform monitoring 100,000+ homes needs live consumption data.

    For those cases, we use:

  • Kafka or Kinesis for high-throughput event ingestion
  • PySpark Streaming or Lambda for stateful processing
  • DynamoDB or Redis for low-latency reads
  • Grafana + Prometheus for operational dashboards with sub-second refresh

    • But these architectures are expensive to build, expensive to operate, and expensive to hire for. We only recommend them when the business requirement genuinely demands sub-second latency. Not "it would be nice" — but "we lose money every second this is delayed."

    The hybrid approach

    The best data platforms we've built use both patterns — streaming for operational metrics that need immediate action, batch/micro-batch for analytical workloads that need depth over speed.

    For a logistics company operating across 100+ countries, we built:

  • Real-time operational dashboards for shipment tracking and alerts (Grafana + Prometheus)
  • 5-minute micro-batch for financial KPIs and P&L reporting (Azure Data Factory + Power BI)
  • Daily batch for deep analytical workloads and ML model training (Microsoft Fabric + Delta Lake)

    • Each tier has different infrastructure, different cost profiles, and different SLAs. Trying to serve all three from a single streaming architecture would have cost 10x more and delivered worse results for the analytical workloads.

    The Airflow sweet spot

    For teams that need something between "Lambda trigger" and "Kafka streaming," Apache Airflow (or Managed Airflow / MWAA) hits the sweet spot. It handles:

  • Scheduled ETL/ELT jobs — every 5, 15, or 60 minutes
  • Dependency management — "don't run this transformation until the source data lands"
  • Retry logic and alerting — failed jobs get retried automatically, engineers get notified
  • Backfill — reprocess historical data when you change transformation logic

    • We've deployed Airflow on projects where the alternative was a pile of cron jobs and bash scripts. The cost of Managed Airflow ($300-500/month) is trivial compared to the engineering time saved debugging failed cron jobs at midnight.

    Making the decision

    When a new project needs analytics, we use this framework:

  • Latency > 1 hour: Batch processing. Airflow + Snowflake. Cheapest, simplest.
  • Latency 1-15 minutes: Micro-batch. EventBridge + Lambda + Snowflake. Serverless, low-cost.
  • Latency < 1 minute: Near-real-time. DynamoDB Streams + Lambda + Redis/Grafana.
  • Latency < 1 second: Streaming. Kafka/Kinesis + PySpark + DynamoDB. Expensive. Make sure you need it.

    • Most products start at micro-batch and never need to move. The ones that do scale into streaming do so gradually — one data pipeline at a time, not a big-bang re-architecture.

    The bottom line

    Real-time analytics is a spectrum, not a binary. The architecture that's right for your latency requirement, your data volume, and your budget is probably simpler and cheaper than you think.

    Start boring. EventBridge, Lambda, Snowflake. Measure whether it's fast enough. In our experience, it almost always is. And when it isn't, you'll know exactly which pipeline needs upgrading — because you measured it, not because you guessed.

    The teams that ship the best analytics aren't the ones with the fanciest streaming infrastructure. They're the ones that matched their architecture to their actual requirements and spent their budget on data quality instead of infrastructure.