Data Element

我干了什么究竟拿了时间换了什么

Spark Structured Streaming Integration With Kafka

Practice track for the kafka integration

Error Messages Met 1 2 3 4 5 6 20/12/09 15:34:53 ERROR Executor: Exception in task 7.0 in stage 0.0 (TID 7) java.lang.IllegalStateException: Cannot fetch offset 356071232 (GroupId: spark-kafka-sour...

Posted by Hanke on December 9, 2020

Shell Command

Shell Daily Help Command

Copy Content from File to Clipboard 1 pbcopy <filename

Posted by Hanke on December 7, 2020

Checkpoint of Spark Structured Streaming

Checkpoint Limitation and Scenarios

Usages in Query This checkpoint location perserves all the essential information that uniquely identifies a query. Hence, each query must have a different checkpoint location, and multiple queries ...

Posted by Hanke on December 6, 2020

Markdown Practice

Learn how to use the markdown frequent grammar

It will record the daily markdown grammar. Title 1 2 3 4 5 6 # First ## Second ### Third #### Fourth ##### Fivth ###### Sixth Fonts RUNBOOK.COM GOOGLE.COM italic text bold text bold and italic ...

Posted by Hanke on December 3, 2020

Jekyll to Setup Github Pages

Setup your own blog

This is the track for local setup for the github pages. Before that suppose you already have the github.io local github repository. For Jekyll case , please refer the guideline. Prepare Env Insta...

Posted by Hanke on December 2, 2020

Spark StreamingQueryListener

Customized Spark Streaming Query Listener Implementation

This blog will explain the spark streaming query listener interface, and how to implement one customized listener for steaming cases. What’s more, how to call it in the streaming job. Listener Int...

Posted by Hanke on December 1, 2020

Spark Join

Details of the spark join

This document will explain the DB join and spark join. [TOC] DB Join SQL Join Types Inner or Outer Joins EQUI Join NON EQUI Join (Theta Join) Join Implementation Nested Loop Join Hash Join Sort M...

Posted by Hanke on September 22, 2020