Behold, a #GitHub page promising to solve "real-world problems" with stream processing demos—because nothing screams practicality like endless code snippets that send developers into a #debugging abyss. Meanwhile, Copilot will 'help' by auto-generating more bugs, proving that #AI is the future... of creating jobs in tech support.
https://github.com/risingwavelabs/awesome-stream-processing #StreamProcessing #TechSupport #CodeSnippets #HackerNews #ngated

#StreamProcessing with #KafkaStreams

Do you know SQL? Exactly!

Most databases and data processing tools support SQL for exactly that reason. And we see a strong movement for all of them to get closer to the standard, day by day.

In this weeks episode of the Cloud Commute podcast, our host @noctarius2k talks with @gunnarmorling from #Decodable about the benefits of #SQL, how #CDC (change data capture) works and why Decodable uses #ApacheFlink as the underlying technology for its #StreamProcessing offering.

https://youtu.be/qrWBboOPY5U

FOMO'd on #KafkaSummit? Want to catch up on what's happening in the data and streaming world?

Here is the March edition of #CheckpointChronicle with tons of great content for you.

https://dcbl.link/cc-mar242

Excellent #apachekafka performance analysis blog from Maciej Mościcki and Piotr Rżysko.

Nice methodical use of tcpdump, flame charts and more to pinpoint the issue and work out remedial steps.

https://blog.allegro.tech/2024/03/kafka-performance-analysis.html

Shhh don't tell anyone… actually tell everyone!

#**ApacheFlink 1.19 RC2 **has just dropped. This is a release candidate, prior to the official release once it's signed off by the committers.

Download it! Read the release notes! Try it out (just not in Production)!

Details and links: https://lists.apache.org/thread/x61673q3ro52jvwj137hct2bqowzcy2v

The February edition of **#CheckpointChronicle** is out

It's got tons of content with links out to excellent stuff from folk across the data and streaming landscape

https://dcbl.link/checkpoint-chronicle-6f6a0a

**#dataEngineering** **#streamProcessing** **#apacheFlink** **#apacheKafka** **#RDBMS** **#datadon**

Hands-On with Catalogs in Flink SQL

In this second post in the series, @rmoff shows how to use Flink SQL with catalogs including #apacheHive, #JDBC, & #apacheIceberg. It also includes a closer look at the data structures within the Hive Metastore.

https://dcbl.link/flink-catalogs---2

This is a really useful write up of how Grab evolved their #apacheKafka-based ad-attribution reporting from a Kappa to Lambda architecture, including pros and cons of each.

https://engineering.grab.com/attribution-platform

I always had in mind that the Kappa architecture (https://www.oreilly.com/radar/questioning-the-lambda-architecture/) was something that one evolved *to* from Lambda—and perhaps it's just a terminology difference—so it's interesting to see it the other way around.

Guten Morgen! Heute starten wir mit der Java User Group Switzerland in unser neues Vortragsjahr! Den Anfang macht Fawaz Ghali mit einem Triple in Zürich (16.01.), St. Gallen (17.01.) und Bern (18.01.) mit Vorträgen über #StreamProcessing.

Ebenfalls am Donnerstag (18.01.) wird uns @maeddes in Luzern etwas über #Java Entwicklung mit Cloud-basierten IDEs erzählen.

Bitte meldet euch an, damit wir für ausreichend Snacks und Drinks sorgen können: https://www.jug.ch/

Last month @gunnarmorling and I launched #CheckpointChronicle, a roundup of interesting things in the streaming & data space.

The next edition is out soon—stay tuned to the @decodable blog, or read last month's and subscribe to receive it by email at https://www.decodable.co/blog/checkpoint-chronicle-november-2023

(scroll down to the bottom and leave your email address. we'll only use it for sending you the newsletter each month, promise )

My learning #apacheflink blog series continues, with a poke around options for connecting to Flink with JDBC: https://rmoff.net/2023/11/16/learning-apache-flink-s01e06-the-flink-jdbc-driver/

It might be 45 sleeps until Christmas, but it's only FOUR DAYS until @gunnarmorling and I launch our monthly roundup of interesting things happening in the streaming and data space. Stay tuned to the Decodable blog next week :)

https://decodable.co/blog

A nice collection of real-world production uses of @ApacheKafka and #ApacheFlink, collected and curated by Thanh Tung Dao :
Flink: https://github.com/dttung2905/flink-at-scale
Kafka: https://github.com/dttung2905/kafka-in-production

#ApacheFlink 1.18 release notes: https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/ with tons of new goodies, including …

Time Travel! https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/time-travel/
JDBC driver for the SQL gateway https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/#introduce-flink-jdbc-driver-for-sql-gateway
Stored Procedure support for catalogs https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/procedures/ (already implemented in #ApachePaimon)
Flink SQL performance improvements https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/#performance-improvements--tpc-ds-benchmark
SQL Client "Quality of Life" improvements (DX) https://cwiki.apache.org/confluence/display/FLINK/FLIP-189%3A+SQL+Client+Usability+Improvements

I've been having some fun (and not so much fun) poking around with Flink SQL for the first time. Lots to get my head around quite a surface area of APIs and concepts to grok.

The client has different ways to display query results, configured by the result-mode option. These include an interactive table, and something much more akin to other SQL CLIs like psql etc.

Good morning #copenhagen!

Come join me and David Moravek tomorrow night for a double header: a deep dive into the #apacheKafka producer request lifecycle and an introduction to #streamProcessing with #apacheFlink.

You won't want to miss it!
https://www.meetup.com/copenhagen-apache-kafka-meetup-by-confluent/events/296247288/

Learn how #ChangeDataCapture & #StreamProcessing can help developers with typical challenges they often face when working on #microservices!

Gunnar Morling talks about the challenges you might encounter when deploying CDC into practice: https://bit.ly/46EjUGs

#InfoQ video w/ #transcript included

What *is* **#ApacheFlink**?

In this second post in my **#learningFlink** series, I'm taken a high-level look at what it is, where you can run it, who uses it, and the kind of use cases for which it's designed.

https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/

(and if you missed it, here's part 1 of the series: https://rmoff.net/2023/09/29/learning-apache-flink-s01e01-where-do-i-start/)

**#dataEngineering** **#streamProcessing** **#learningInPublic** **#alwaysBeLearning #openSource #datadon**

Learning #ApacheFlink S01E01: Where Do I Start? https://rmoff.net/2023/09/29/learning-apache-flink-s01e01-where-do-i-start/