Everday syslog-ng: How to configure syslog-ng Kafka destination to use Kerberos

Hello guys,

Now I just want to write something interesting about my favorite logging application called syslog-ng.

Prerequisites

  • Active Kerberos server
  • Active Apache Kafka server configured to use Kerberos
  • The Kerberos client libs (krb5-user krb5-config) are installed and configured on the host where syslog-ng is running
  • syslog-ng OSE 3.12 or newer
  • or syslog-ng Premium Edition 7.0.5 or newer
  • Java Cryptography Extension (JCE): To use Kerberos, syslog-ng needs this Java extension. You should download and install it to your computer where syslog-ng runs.

In the first scenario, I will configure syslog-ng to use Kerberos with SASL_PLAINTEXT authentication, in the second part, Kerberos with SSL.

Syslog-ng configuration

First of all we need a syslog-ng configuration. Here is mine (the embedded configs are detailed below):

@version: 3.12
@module “mod-java”
@include “scl.conf”
options {
jvm_options(“-Djava.security.auth.login.config=/home/pzolee/install/configs/kafkakerberos/client.jaas -Djava.security.krb5.conf=/etc/krb5.conf -Dlog4j.configuration=file:/home/pzolee/install/configs/kafkakerberos/log4.properties”);
};

source s_network_5acd1de3c3494308820dd80525d0a1aa {
network(ip(“127.0.0.1”)
port(10001));
};

destination d_kafka_e0fb1bbb06d345f4ba3770b4eecc4a8a {
kafka(
client_lib_dir(/home/pzolee/install/kafka_2.11-0.11.0.1/libs)
kafka-bootstrap-servers(“kafkabrokers.mydomain:9092”)
topic(“test_topic”)
properties-file(“/home/pzolee/install/configs/kafkakerberos/producer.properties”)
);

};

log {
source(s_network_5acd1de3c3494308820dd80525d0a1aa);
destination(d_kafka_e0fb1bbb06d345f4ba3770b4eecc4a8a);
flags(flow-control);
};

It just contains a simple network (tcp) source where the logs are coming, and a Kafka destination where the logs are forwarded. For the options of Kafka destination, read the documentation.

The client.jaas file contains the required parts for Kerberos:

cat client.jaas
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required refreshKrb5Config=true useKeyTab=true keyTab=”/home/pzolee/install/configs/kafkakerberos/client.keytab” storeKey=true principal=”client/thor.mydomain@SYSLOG-NG.MYDOMAIN”;};

Keytab file

Because syslog-ng runs as a daemon without interactive user login, we want to use keytab file instead of manually initialize Kerberos and typing the password.

So generate a Kerberos keytab file for syslog-ng. Here is the example of my keytab file:

klist -kte /home/pzolee/install/configs/kafkakerberos/client.keytab
Keytab name: FILE:/home/pzolee/install/configs/kafkakerberos/client.keytab
KVNO Timestamp           Principal

—- ——————- ——————————————————

1 2017-09-26 15:18:23 client/thor.mydomain@SYSLOG-NG.MYDOMAIN(aes256-cts-hmac-sha1-96)

Don’t forget, Kerberos checks the FQDN in the principal, so the hostname should be correct.
Properties files

We need a Kafka producer properties files for syslog-ng with the following content:

cat /home/pzolee/install/configs/kafkakerberos/producer.properties
bootstrap.servers=kafkabrokers.mydomain:9092
compression.type=none
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

For bootstrap.servers you should use your Kafka server and for sasl.kerberos.service.name the same name that is configured in the server.properties file in the Kafka server.

And another one for log4j to see some debug messages

cat /home/pzolee/install/configs/kafkakerberos/log4.properties
log4j.rootLogger=INFO

Now start syslog-ng:

sbin/syslog-ng -Fevd

[2017-09-28T10:24:08.306159] syslog-ng starting up; version=’3.11.1′
[2017-09-28T10:24:08.341826] ProducerConfig values: \x0a acks = 1\x0a batch.size = 16384\x0a bootstrap.servers = [thor.mydomain:9092]\x0a buffer.memory = 33554432\x0a client.id = \x0a compression.type = none\x0a connections.max.idle.ms = 540000\x0a enable.idempotence = false\x0a interceptor.classes = null\x0a key.serializer = class org.apache.kafka.common.serialization.StringSerializer\x0a linger.ms = 0\x0a max.block.ms = 60000\x0a max.in.flight.requests.per.connection = 5\x0a max.request.size = 1048576\x0a metadata.max.age.ms = 300000\x0a metric.reporters = []\x0a metrics.num.samples = 2\x0a metrics.recording.level = INFO\x0a metrics.sample.window.ms = 30000\x0a partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner\x0a receive.buffer.bytes = 32768\x0a reconnect.backoff.max.ms = 1000\x0a reconnect.backoff.ms = 50\x0a request.timeout.ms = 30000\x0a retries = 0\x0a retry.backoff.ms = 100\x0a sasl.jaas.config = null\x0a sasl.kerberos.kinit.cmd = /usr/bin/kinit\x0a sasl.kerberos.min.time.before.relogin = 60000\x0a sasl.kerberos.service.name = kafka\x0a sasl.kerberos.ticket.renew.jitter = 0.05\x0a sasl.kerberos.ticket.renew.window.factor = 0.8\x0a sasl.mechanism = GSSAPI\x0a security.protocol = SASL_PLAINTEXT\x0a send.buffer.bytes = 131072\x0a ssl.cipher.suites = null\x0a ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]\x0a ssl.endpoint.identification.algorithm = null\x0a ssl.key.password = null\x0a ssl.keymanager.algorithm = SunX509\x0a ssl.keystore.location = null\x0a ssl.keystore.password = null\x0a ssl.keystore.type = JKS\x0a ssl.protocol = TLS\x0a ssl.provider = null\x0a ssl.secure.random.implementation = null\x0a ssl.trustmanager.algorithm = PKIX\x0a ssl.truststore.location = null\x0a ssl.truststore.password = null\x0a ssl.truststore.type = JKS\x0a transaction.timeout.ms = 60000\x0a transactional.id = null\x0a value.serializer = class org.apache.kafka.common.serialization.StringSerializer\x0a;
[2017-09-28T10:24:08.713300] Successfully logged in.;
[2017-09-28T10:24:08.716513] [Principal=client/thor.mydomain@SYSLOG-NG.MYDOMAIN]: TGT refresh thread started.;
[2017-09-28T10:24:08.719814] [Principal=client/thor.mydomain@SYSLOG-NG.MYDOMAIN]: TGT valid starting at: Thu Sep 28 10:24:21 CEST 2017;
[2017-09-28T10:24:08.720000] [Principal=client/thor.mydomain@SYSLOG-NG.MYDOMAIN]: TGT expires: Thu Sep 28 20:24:21 CEST 2017;
[2017-09-28T10:24:08.720146] [Principal=client/thor.mydomain@SYSLOG-NG.MYDOMAIN]: TGT refresh sleeping until: Thu Sep 28 18:29:04 CEST 2017;

As you can see, syslog-ng could log in to Kafka using kerberos.

Configuring syslog-ng Kafka destination with Kerberos and SSL

Prerequisites

  • Kafka broker is configured to use Kerberos with SSL

You only need a few minor changes in producer.properties file:

cat /home/pzolee/install/configs/kafkakerberos/producer.properties
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
security.protocol=SASL_SSL
ssl.keystore.location=/home/pzolee/install/configs/kafkakerberos/servicestore.jks
ssl.keystore.password=some_keystore_pass
ssl.key.password=some_key_pass
ssl.truststore.location=/home/pzolee/install/configs/kafkakerberos/truststore.jks
ssl.truststore.password=some_truststore_pass

You only have to change the security.protocol from SASL_PLAINTEXT to SASL_SSL and provide the keystore and truststore paths and passwords.

If you want to see the ssl logs, add “-Djavax.net.debug=all” to the jvm_options.

Advertisements

How to test your NAS speed

This is just something personal.

Recently I installed a Synology NAS (DS216j) to my home and wanted to test the speed of this NAS. Of course you can test it via file copy but I wanted to know several values:

  • the maximum available raw speed when my computer is connected to the NAS
  • the maximum available raw speed when my mobile is connected to the NAS
  • and the samba speed when my mobile using the NAS

So first of all, I enabled ssh connection to my NAS, then copied my wifi-speed-test-server.py file to the NAS. This Synology NAS already contains python support, so I could start it easily:

pzolee@pzoleenas:/volume1/$ python wifi_speed_test_server.py

Then just downloaded and started the java version of wifi speed test app, set the IP address of the NAS and started the test:

Wow, it was impressive 316 Mbps (~40 MB/s). My laptop is using AC wifi.

Then I tested the same on my mobile (LG G4) with my WiFi Speed Test app:

 

It was very similar, 33 MB/s.

And finally I tested the samba speed:

Using my laptop, I could reach the 40 MB/s transfer rate via samba, but my phone could only transfer with 7 MB/s.

What you have to know about SD card speed

Hi guys,

I just want to discuss something related to my SD Card Test app. Sometimes I got negative feedback (low rating) with the following content:

“this app is wrong because it reports slow sd card speed (only 20 MB/s) and I have a Class 10 sd card subtitled up to 100 MB/s”

In fact, the app is good and the sd card is slow, that’s the reality.

Class 10 means the minimum sequential write speed is 10 MB/s. That’s all and nothing more. Anything above this value is just “up to”, and in fact that means nothing.

Let me tell you an example:

I have an Samsung EVO 64 GB Sd card, the type is Class 10/U1/XC1.  The spec mentions “up to” 48 MB/s.

But is this true? The trick is the next: the highlighted value is the read speed and not the write.

The reality is the next:

  • write speed:  23 MB/s
  • read speed: 45 MB/s

I tested it with my app, and with my computer too, the result is the same.

My app will assign the class type based on the write speed and you can compare the read and write speed.

 

 

 

 

 

 

 

 

 

The specification is the next:

  • class 10/UHS-1: the minimum sequential write speed is 10 MB/s
  • UHS-3/V30: the minimum sequential write speed is 30 MB/s
  • V60: the minimum sequential write speed is 60 MB/s
  • V90: the minimum sequential write speed is 90 MB/s

So when you find the measured speed too slow, don’t think my app is wrong, just check the class type and don’t believe for the marketing texts (“up to”).

SD Card Test

 

device-2017-01-17-222230Hi guys,

I have a new app on Google Play, called SD Card Test. You can download it using this link.

Using this app, you can test the speed of the internal (built-in) or external (removable) SD card. You can measure both read and write speeds.

 

 

 

 

Here are a few more screenshots:

device-2017-01-17-222325device-2017-01-17-222305

Network Signal Monitor for Android

Hello guys!

networksignalmonitor1Let me introduce my new app called Network Signal Monitor. You can download it from Google Play, link here.

This app is very similar to my existing WiFi Analyzer application, but it monitors and analyzes the cellular network of your mobile phone. So it will check your network type like LTE or HSDPA, the latency, signal strength and others.

Main features:
* detailed mobile network information
* detailed SIM information
* analyzing function
* real-time signal strength on the dashboard and on the graph.

As this is a new app, forgive me if you find any error or problem in the app, but feel free to send me your remarks.

Dual SIM devices are also supported, but currently only partially (I’m working on it).

More screenshot about the application:

networksignalmonitor2networksignalmonitor3

Mobile Network Glossary

This is an online help for the Network Signal Monitor app.

Signal strength: This is the power of the received signal. The weakest the worst. The acceptable value depends on the given network type. For example -90 dBm is good for an LTE network but low for a GSM network.

MCC: Mobile country code. The mobile country code consists of 3 decimal digits defined by E.212. More details are here.

MNC: Mobile network code. The mobile network code consists of 2 or 3 decimal digits defined by E.212. More details are here.

ASU: Arbitrary Strength Unit (ASU) is an integer value proportional to the received signal strength measured by the mobile phone. More details are here.

IP address: An Internet Protocol address (IP address) is a numerical label assigned to each device (e.g., computer, mobile phones) participating in a computer network that uses the Internet Protocol for communication. Example: 192.168.0.1

latency: This is the latency time of the server in ms.

Disable Adaptive Brightness in Windows 10 / LG monitor

Hi guys,

After I installed Windows 10 to my new computer, I experienced a very annoying thing: the brightness level of the screen went up and down without my intent… I thought it is caused by Windows 10 (but it was only half-true).
Ok, if you google this thing, you will find a lot of articles about how to disable adaptive brightness in Windows 10. Basically go to Control Panel -> Power Options -> Change plan settings -> Advanced and then at the display settings disable Adaptive Brightness.

This did not work…

After reading a tons of forum, I found that my computer has Intel HD 520 VGA and it also contains this feature. So I had to do the following:

Open Intel HD Graphics Control Panel -> Graphics Properties -> Power -> On battery and disable Display Power Saving Technology.

And it still did not work, the brightness of the screen always changed. Very-very frustrating…

Finally, I found the reason of the problem and the solution.

 

20160923_210153My new LG monitor also knows this “feature”. It’s called as “Smart Energy Saving“.
So the solution: go to monitor menu -> Quick Settings and disable “Smart Energy Saving” function. By default it was set to “Low”. After I turned it off, the problem is gone. No more brightness change excepting if I am changing it.

Hint: some monitor calls this feature as DCR (Dynamic Contrast Ratio).

20160923_210211

Disable this function