Structured Streaming

Structured Streaming is the Apache Spark API that lets you express computation on streaming data in the same way you express a batch computation on static data. The Spark SQL engine performs the computation incrementally and continuously updates the result as streaming data arrives. For an overview of Structured Streaming, see the Apache Spark Structured Streaming Programming Guide.

These topics provide introductory notebooks, details on how to use specific types of streaming sources and sinks, how to put streaming into production, and notebooks demonstrating example use cases:

• Introductory Notebooks
  • Structured Streaming Demo in Python
  • Structured Streaming Demo in Scala
• Streaming Data Sources and Sinks
  • Apache Kafka
  • Azure Event Hubs
  • Databricks Delta Tables
  • Read and Write Streaming Avro Data with DataFrames
  • Write to Arbitrary Data Sinks
  • Optimized Azure Blob Storage File Source with Azure Queue Storage
• Structured Streaming in Production
  • Recover from query failures
  • Configure Spark scheduler pools for efficiency
  • Optimize performance of stateful streaming queries
  • Multiple watermark policy
• Examples
  • Write to Cassandra using foreachBatch() in Scala
  • Write to Azure SQL Data Warehouse using foreachBatch() in Python
  • Stream-Stream Joins

For detailed information on how you can perform complex streaming analytics using Apache Spark, see the posts in this multi-part blog series:

• Real-time Streaming ETL with Structured Streaming
• Working with Complex Data Formats with Structured Streaming
• Processing Data in Apache Kafka with Structured Streaming
• Event-time Aggregation and Watermarking in Apache Spark’s Structured Streaming
• Taking Apache Spark’s Structured Streaming to Production
• Running Streaming Jobs Once a Day For 10x Cost Savings: Part 6 of Scalable Data
• Arbitrary Stateful Processing in Apache Spark’s Structured Streaming