Kafka Connect in Action: Loading a CSV file into Kafka



Kafka Connect in Action: Loading a CSV file into KafkaKafka Connect in Action: Loading a CSV file into KafkaKafka Connect in Action: Loading a CSV file into Kafka

  • Name: Kafka Connect in Action: Loading a CSV file into Kafka
  • Duration: 43m 21s
  • Uploader: Robin Moffatt
  • Added On: 19 June, 2020
  • Views: 3375
  • Likes: 53
  • Dislikes: 1
  • Video Source: YouTube



Ingesting data from a CSV file into a Kafka topic can be done easily using Kafka Connect. Kafka Connect is part of Apache Kafka, and only requires a JSON file to configure - no coding! This video shows you how to stream CSV files into Kafka, how to manage the schema, and as a bonus also shows streaming the same data out to a database such as Postgres, as well as processing it with ksqlDB. To learn more about Kafka Connect see https://docs.confluent.io/current/connect/index.html?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_N1pseW9waNI&utm_term=rmoff-devx Code used in this video: https://github.com/confluentinc/demo-scene/tree/master/csv-to-kafka -- ℹī¸ Table of contents 00:00:44 Brief introduction to Kafka Connect 00:01:39 Checking that the correct Kafka Connect plugin is installed 00:02:46 Kafka, Bytes, and Schemas 00:05:06 Creating a connector to ingest data from CSV file and set the schema based on header row 00:09:43 File metadata stored in the Kafka message header 00:10:18 Setting the message key 00:14:35 Manipulating the schema / changing data types 00:19:22 Ingesting raw CSV into a Kafka topic without a Schema 00:22:43 Streaming data from CSV into Kafka into a Database 00:28:27 Updating database rows in-place from CSV file ingest 00:28:42 Impromptu Kafka Connect troubleshooting ;-) 00:32:34 Filtering and aggregating CSV data in Kafka using ksqlDB -- 💾 Download Kafka Connect Spooldir plugin: https://www.confluent.io/hub/jcustenborder/kafka-connect-spooldir 🎈 Kafka Connect Spooldir docs: https://docs.confluent.io/current/connect/kafka-connect-spooldir/connectors/csv_source_connector.html?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_N1pseW9waNI&utm_term=rmoff-devx 🎈 Single Message Transforms (SMT): https://docs.confluent.io/current/connect/transforms/index.html?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_N1pseW9waNI&utm_term=rmoff-devx 🎈 Cast SMT: https://docs.confluent.io/current/connect/transforms/cast.html#cast?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_N1pseW9waNI&utm_term=rmoff-devx 🎈 Kafka Connect JDBC Source connector: https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_N1pseW9waNI&utm_term=rmoff-devx 🎈 Kafka Connect JDBC Sink connector docs: https://docs.confluent.io/current/connect/kafka-connect-jdbc/sink-connector/sink_config_options.html#data-mapping?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_N1pseW9waNI&utm_term=rmoff-devx đŸŽĨ Installing JDBC Driver for Kafka Connect: https://rmoff.dev/fix-jdbc-driver-video đŸŽĨ Streaming data from Kafka to a database: https://rmoff.dev/kafka-jdbc-video đŸŽĨ JDBC sink and schemas: https://rmoff.dev/jdbc-sink-schemas đŸŽĨ Introduction to ksqlDB: https://rmoff.dev/ksqldb-introduction -- ☁ī¸ Confluent Cloud ☁ī¸ Confluent Cloud is a managed Apache Kafka and Confluent Platform service. It scales to zero and lets you get started with Apache Kafka at the click of a mouse. You can signup at https://confluent.cloud/signup?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_N1pseW9waNI&utm_term=rmoff-devx and use code 60DEVADV for $60 towards your bill (small print: https://www.confluent.io/confluent-cloud-promo-disclaimer/)

Download Kafka Connect in Action: Loading a CSV file into Kafka in Full HD MP4 3GP MKV Video and MP3 Torrent

Download Kafka Connect in Action: Loading a CSV file into Kafka in Hd, Download Bangla, English, Hindi, Tamil Kafka Connect in Action: Loading a CSV file into Kafka Full Mp3 Album Songs Bangla , Download Full HD Kafka Connect in Action: Loading a CSV file into Kafka Video Songs , Bangla Romantic, Download Kafka Connect in Action: Loading a CSV file into Kafka HD Bangla Video Songs Free , Love, Funny, Comedy, Hasir Natok Kafka Connect in Action: Loading a CSV file into Kafka Natok full HD PC Mp4 3gp 720p, 1080p Download Now , Download Kafka Connect in Action: Loading a CSV file into Kafka Bollywood Movie Videos , Download Kafka Connect in Action: Loading a CSV file into Kafka Kolkata Bangla Video Songs, Download New Unrelesed Kafka Connect in Action: Loading a CSV file into Kafka Bangla Full Mp3 Songs 192 kbps 64 kbps, Download New Unrelesed Kafka Connect in Action: Loading a CSV file into Kafka Bollywood Hindi Full Mp3 Songs 192 kbps 64 kbps , Download Kafka Connect in Action: Loading a CSV file into Kafka English Romantic, Vevo HD Video Songs , Download Kafka Connect in Action: Loading a CSV file into Kafka Telugu Tamil Full Mp3 Songs 192 kbps 64 kbps , Download New Unrelesed Kafka Connect in Action: Loading a CSV file into Kafka HollyWood English Movie Full Mp3 Songs 192 kbps 64 kbps, Download, Kafka Connect in Action: Loading a CSV file into Kafka full Movie Download uTorrent , Download New Unrelesed Kafka Connect in Action: Loading a CSV file into Kafka full album zip file CD rip download , Kafka Connect in Action: Loading a CSV file into Kafka full Movie Download kickass torrent 1080p HD , Download Kafka Connect in Action: Loading a CSV file into Kafka Kolkata Bangla, Bengali Full HD Movie , Download Kafka Connect in Action: Loading a CSV file into Kafka Full Bangla HD Movie , Download Kafka Connect in Action: Loading a CSV file into Kafka Hollywood English Movie , Download Kafka Connect in Action: Loading a CSV file into Kafka BollyWood Full HD Movie Watch, Now , Download Kafka Connect in Action: Loading a CSV file into Kafka Hindi, Tamil Movie , Download, Bangladesh, India, Pakistan MP3, 3GP, MP4, HD, MKV, Avi Kafka Connect in Action: Loading a CSV file into Kafka (2017) all video Free Download , master print download full movie , Download Kafka Connect in Action: Loading a CSV file into Kafka clear print download, Download Kafka Connect in Action: Loading a CSV file into Kafka full movie official print download DVDrip Vcdscam webrip Dvdscam download now

Related Videos


© Video24.Mobi 2021
All Rights Reserved.