155 questions
Best practices
0
votes
2
replies
44
views
Kafka Topic Architecture for 200+ IoT Devices: One Topic per Device vs. Shared Topic?
I am designing a system using .NET that acts as a MES system between 200+ pieces of equipment (IoT devices). The flow is bidirectional:
Equipment -> My .NET System -> Equipment
I need low ...
- 13
0
votes
0
answers
82
views
Kafka Producer throws exception when initiating transaction
I have a .NET application that uses the Confluent client library to send messages to Kafka with a transactional producer. It uses a pool of producers to avoid threading issues with transactions, and ...
- 4,890
0
votes
0
answers
16
views
Kafka producer timeout before stop trying to send
I am trying to use DeliveryReport handler with Produce. I want to simulate that there is a Kafka problem by tuning off the Kafka server so delivery fails.
However I was expecting to get a ...
- 83
0
votes
2
answers
494
views
How to prevent my Kafka consumer from consuming new messages
I’m working with two Kafka topics. The first topic is consumed by a continuous consumer that processes messages as soon as they arrive. If the processing fails for any message, that message is ...
- 460
1
vote
0
answers
52
views
Uneven distribution of consumed messages per consumer inside same consumer group for Kafka
I have to consume messages from a topic on a Kafka cluster with 10 partitions where messages are being evenly distributed accros them. I have a ASP.NET application with 10 background services where ...
-1
votes
1
answer
1k
views
KAFKA Security error - SASL PlainText error
i am new to kafka and my requirement is to implement SASL PLAINTEXT (username/ password) security in kafka broker(s). i tried different things but on success till now. what i have confs i did were
...
- 85
0
votes
1
answer
25
views
Kafka Consumers leave some messages unread
Multiple consumers sharing the same ConsumerGroupId read data from multiple topics (from multiple threads if that matters). All topics have multiple partitions, AutoCommit is set to false for all ...
- 111
1
vote
1
answer
171
views
Azure Event Hubs with Confluent.Kafka. I can produce but not consume
I'm struggling with an issue that I cannot understand.
With this configuration, I can successfully produce a message;
var config = new ProducerConfig
{
BootstrapServers = bootstrapServers,
...
- 1,820
0
votes
1
answer
116
views
How to always read from the end of the topic while using Confluent .NET API
I have a .NET Consumer of a Kafka topic that uses the Confluent .NET API. I want to always read from the end of the topic.
For example, if the Consumer disconnects for a while and then reconnects, I ...
- 125
0
votes
0
answers
74
views
How to setup De-Serializer for Kafka Streams Application when message was produced in .NET and has type Confluent.Kafka.Message<TKey, TValue>
I am producing a message to a topic "ITEM_PRICES" using dontnet(Confluent.kafka) library). This Message is meant to be a Key,Value pair with the Key as a string and value as float. I setup ...
0
votes
0
answers
36
views
How to convert .NET DateTime to kafka Timestamp & Date types [duplicate]
I need to convert .NET DateTime fields to the corresponding Kafka Java types as declared on the Java library before producing messages to Kafka.
How to convert a .NET DateTime to Kafka Timestamp & ...
- 180
0
votes
1
answer
241
views
librdkafka producer configuration to control connection towards Kafka broker
Which configuration parameter controls the number of connections the librdkafka producer opens toward the kafka broker?
I use librdkafka for producing the message into the kafka. I discovered it uses ...
- 109
1
vote
0
answers
340
views
Random test failures in kafka integration tests with TestContainer
I'm experiencing inconsistent failures in my Kafka integration tests using TestContainers. The tests involve simple produce and consume operations along with database seeding, but sometimes they pass ...
0
votes
0
answers
225
views
Confluent Kafka - Transaction Coordinator not running
My team and I are relatively new to Kafka and have had some success running kafka locally via a docker compose file and podman.
One of my colleagues created the compose file, configuring the config ...
- 587
1
vote
0
answers
281
views
Confluence Kafka Producer and consumer client Unit Tests
.NET 6
Confluent.kafka v1.9.3
I have created a Confluent Kafka Consumer client application to receive messages from topic which is working as expected.
Now I am trying to create Nunit test project for ...
- 189
1
vote
0
answers
27
views
used kafka in unty, but force quit
I can't speak English. Sorry in advance.
i used kafka consume in uniyt. and i want safe shutdown kafka. so i used Close() or Dispose() in OnApplicationQuit(), but unity not work and force quit.
using ...
- 11
1
vote
0
answers
571
views
When I subscribe to a Kafka topic, it takes too long until I get first result. Afterwards the consuming cycle is normal. What could be the problem?
I am using following simple code (based on confluent Kafka) to connect and subscribe to a Kafka topic. After I call this method, it takes up to 5-10 seconds until I get the consume result. The server ...
- 801
1
vote
2
answers
1k
views
Consuming messages from Kafka topic one by one takes too long time. How can I shorten this time? Is reading multiple messages at one time possible?
I am using Confluent Kafka library for consuming data from a Kafka topic with the code below. I am consuming the messages one by one in a while loop, staring with the last message. (Previous messages ...
- 801
0
votes
1
answer
865
views
How can I reset the topic offset while consuming from Kafka topic?
I start to consume from the last message of a Kafka topic with following code. (Confluent Kafka). What I want to do is:
I want to consume the last 50 messages of a Kafka topic
wait for 5 seconds and ...
- 801
1
vote
1
answer
2k
views
How can I convert the Kafka message timestamp to date and time format in C#?
In my C# code I programmed a consumer with Confluent Kafka. I am reading the timestamp of the message and want to convert it to the know time and date format. I tried following code but get always ...
- 801
1
vote
1
answer
2k
views
How to implement parallel task consumer for kafka application?
I want to consume my kafka topic using multiple tasks in task parallel library.
var tasks = new List<Task>();
consumer.Subscribe(topicName);
var capacity = 5;
var counter = 0;
while (true)
{
...
- 6,965
0
votes
0
answers
123
views
Kafka Consumer error: System.InvalidOperationException: FATAL: Cannot marshal from a NULL ptr
Kafka Consumer error: System.InvalidOperationException: FATAL: Cannot marshal from a NULL ptr .
Any idea why this error is happening? This is with confluent kafka dot net nuget library.
Some cases ...
- 1
1
vote
1
answer
408
views
MassTransit Kafka and Outbox: Getting No such configuration property: "schema.registry.url" Exception
I have setup MassTransit with Kafka and Outbox pattern
var confluentConfiguration = builder.Configuration.GetSection("Kafka").ToConfluentConfiguration();
builder.Services.AddSingleton<...
- 11k
0
votes
0
answers
111
views
Why first message in recordList always goes in if block "deliveryReport.Status == PersistenceStatus.NotPersisted"?
I am trying to store batch of messages that failed to publish to kafka in DB and have added a callback deliveryReport for the same. However, when broker is up and running and there is no other issue, ...
0
votes
1
answer
235
views
My Kafka Topic has three partitions. I cyclic read from all partitions the newest data coming from source, but one reading cycle takes too long. Why?
In my usecase tha data source (CNC machine) is sending data (flow value of a liquid) to a Kafka topic in one second interval. There a machanism in the source side that provides that the data is send ...
- 801
0
votes
1
answer
1k
views
How can I read always the newest (last) message from a Kafka Topic that has more then one partition (In my case three partitions)?
In my .net C# project (with Confluent Kafka library) currently I am using following code to read the newest message from a Kafka topic. But with this code I can read the newest message from the ...
- 801
-1
votes
1
answer
278
views
How to define custom subject name strategy in dotnet Confluent Kafka
Currently Confluent Kafka dotnet supports three subject name strategy:
Topic Name
Record type
TopicRecord type
I want to create a custom subject strategy where I can generate subject & schema ...
- 61
0
votes
1
answer
281
views
Frequent rebalancing of Apache Kafka freezes the operation for the duration of the rebalancing
When sending a message to Kafka there is a rebalancing that freezes the process of Kafka for a time equal to the balancing, there are logs in which the error is indicated. How to fix it?
[2023-09-25 ...
0
votes
0
answers
844
views
Azure Function KafkaTrigger (.NET) consuming messages with Avro array schema as root
I have an Azure Function KafkaTrigger (using azure functions kafka extension: https://github.com/Azure/azure-functions-kafka-extension) which is set to consume messages from Kafka with Avro schema ...
- 436
0
votes
1
answer
3k
views
Apache Kafka time difference between when a message is consumed and produced
We have a topic with 6 partitions. Traffic is not excessive by Kafka terms. We are using a 3-broker cluster in AWS MSK. We are using the Confluent Kafka package for .NET.
When we consume the topic, ...
- 589
1
vote
0
answers
863
views
Kafka C#: Error occured: Subscribed topic not available: unknown topic or partition
I am trying to run producer/consumer/ktable code in kafka and I keep getting this error message even though the topics are the same. It was working 2 days ago and I haven't touched my code since but ...
- 11
1
vote
1
answer
565
views
Confluent Kafka: How to replicate partitioner hashing function so that it's possible for the consumer to consume the correct partition?
I have set up a Kafka server with 1000 partitions and 1 Topic. Basically, the data that gets stored in the cluster are status updates of objects. There are about 17000 of these objects and I produce ...
- 45
3
votes
2
answers
2k
views
Error producing message Local Value serialization error in Kafka
Hi I am working in Kafka and .Net. I am trying to produce message as below. I have created configuration for schema registry and producer as below. I am using certificates to connect to schema ...
- 351
0
votes
0
answers
312
views
Kafka Consumer does not receive data
I have worked out a Kafka concept which is built up in such a way that I have a series of machines which have electric airpressure dustextraction and machinenStatus data. I have set up my Kafka ...
1
vote
1
answer
4k
views
Simple Confluent Kafka .net client fails to connect to simple build of AK Kafka broker
Any hints on how to get a simple .Net Kafka client to connect to a broker?
I built Apache Kafka from the 3.4.0 download and followed the quick start successfully (topic created, events produced, ...
- 411
0
votes
1
answer
4k
views
How do I make the C# Kafka client consume the latest topic message?
I created a C# Kafa client with Confluent.Kafka, and I have been trying to read the latest message only from two different topics, but it reads all messages, not just the latest one.
Here is my kafka ...
- 372
0
votes
0
answers
368
views
How to come out of while loop once all messages consumed in kafka?
net core console application. I have below implementation of kafka consumer
var config = new ConsumerConfig
{
GroupId = groupId,
BootstrapServers = brokerList,
SaslMechanism = ...
- 351
1
vote
1
answer
1k
views
Confluent Kafka Consumer.Consume() not able to catch any exceptions ( invalid credentials)
using System;
using System.Collections.Generic;
using System.Net;
using System.Security.Authentication;
using System.Threading.Tasks;
using Confluent.Kafka;
class program
{
static async Task Main(...
1
vote
3
answers
6k
views
How can I consume just the last (newest) message from a Kafka topic with Kafka Confluent Consumer?
I have following code for reading data from a Kafka topic. My target is periodically to read just the last newest message in the topic because I want use the data in a live chart. I have written the ...
- 801
0
votes
1
answer
780
views
Kafka producer in C# as Windows Service
I created a Kafka producer, using Confluent library, as a console app that ran as a scheduled task to submit some error records to Kafka broker periodically. Worked fine.
I have to convert this to run ...
- 2,252
0
votes
1
answer
615
views
Confluent Kafka missing methods for .NET
I am using Confluent.Kafka package v2.0.2 that I downloaded via NuGet in a .NET project in order to create a Kafka Consumer and I noticed that Confluent's docs (this page for instance) include many ...
- 395
0
votes
3
answers
6k
views
Authorization issue when performing operations on MSK cluster using SASL/SCRAM auth method
I have configured the MSK cluster and allowed public access through SASL/SCRAM authentication method. Now I am facing the issue where I do not have the necessary permissions when using these ...
- 997
0
votes
1
answer
1k
views
C# Kafka IConsumer's Subscribe method doesn't consume any messages but Assign methods works fine
I'm trying to create a simple Kafka consumer in c# using the below code
private static CancellationTokenSource StartConsumer(IAdminClient client, string topicName)
{
ConsumerConfig config = new()
...
3
votes
1
answer
6k
views
Kafka Producer in .Net - SSL Handshake Failed
This is the first time I am trying to connect to Kafka server using Asp.Net console app and I ran into a bunch of issues. I solved most but can't shake this one off.
I am testing the producer part ...
- 2,252
0
votes
2
answers
1k
views
How to use AvroSerializer without a schema registry
I am trying to write a unit test that verifies that adding a new property to an Avro schema is backwards compatible.
First I took the Avro generated .cs model and saved it as MyModelOld.cs and renamed ...
- 985
0
votes
1
answer
500
views
How to configure a kafka broker in one cluster to publish message
I have two broker and I want to restrict 2nd one with particular messages for particular region.
for example we have multiple region in world to provide our services like china, India
I just wanted to ...
- 23
0
votes
2
answers
8k
views
How to improve Kafka consumer performance
I have a consumer application which I would like to speed up. Most of the time consumer is waiting for http response from the third party system hence I cannot proceed with other messages in queue and ...
- 1,479
1
vote
2
answers
1k
views
Consume ONLY latest message
I'm programming a Kafka Consumer in an ASP.NET web application, all seems good but every time I run the consumer, it consumes all messages in the topic.
Here the Consumer config.
ConsumerConfig = new ...
- 93
2
votes
0
answers
2k
views
Random "Timed out ProduceRequest in flight" messages
I am getting some random timeout errors while publishing messages using Confluent.Kafka. The application runs in a Kubernetes cluster and is built using the .NET 6 framework.
When the default timeout (...
- 41
0
votes
0
answers
670
views
Kafka consumer can't read from multiple partitions at the same time
I use Confluent.Kafka 1.9.2 C# library to create single Kafka consumer for listening topic with several partitions. Currently consumer drain out all messages from first partition and only then goes to ...
