Authorizing Kafka access over non-authenticated channel via Ranger
This section answers some questions one is likely to encounter when trying to authorize access to Kafka over non-authenticated channel. This Kafka feature is available in HDP releases 2.3.4 (Dal-M20) or later.
Can I authorizer access to Kafka over a non-secure channel via Ranger?
Yes. you can control access by ip-address.
Can I authorize access to Kafka over non-secure channel by user/user-groups?
No, one can’t use user/group based access to authorize Kafka access over a non-secure channel. This is because it isn't possible to assert client’s identity over the non-secure channel.
What is a recommended way to set-up policies when trying to control access to Kafka over a non-secure channel?
Ensure that all Brokers nodes have Kafka Admin
access. This is a mandatory step. If you don’t perform this step then your cluster won’t work properly.
-
Identify the nodes where brokers are running.
-
Create a policy where resource is * (i.e. all topics) and grant
Kafka Admin
access type to thepublic
user group. Specify ip-address of all the brokers as theip-range policy condition
on the policy item.
Ensure publishers have appropriate access.
-
Identify ip address of all nodes where publishers would run along with their respective topics.
-
Create policy where resources are the respective topic names and grant
Publish
access type topublic
user group. Specify ip-address of machines where those publishers would run as theip-range policy condition
on the policy item. -
Specify topic name(s) as policy resource. Note that you can specify multiple topics or even regular expressions in topic names.
Ensure consumers have appropriate access. Same process as publishers except change access type to Consume
instead or Produce
.
Why do we have to specify public
user group on all policies items created for authorizing Kafka access over non-secure channel?
-
Kafka can’t assert the identity of client user over a non-secure channel. Thus, Kafka treats all users for such access as an anonymous user (a special user literally named
ANONYMOUS
). -
Ranger's
public
user group is a means to model all users which, of course, includes this anonymous user (ANONYMOUS
).
What are the specific things to watch out for when setting up authorization for accessing Kafka over non-secure channel?
- Make sure that all broker-ips have
Kafka admin
access to all topics, i.e. *. -
Make sure no publishers or consumers are running on broker nodes that need access control. Since broker ips have open access it isn’t possible to control access on those nodes.
I have the policies as specified above, however, I am still not able to consume over an non-authenticated channel using bin/kafka-console-consumer.sh
script that is a part of the Kafka distribution! The consumer hangs and gives the error message “No brokers found in ZK.” What gives?
- Ensure that
/etc/kafka/conf/kafka_client_jaas.conf
does not have specification forserviceName="zookeeper"
. This is typically theClient
section. -
Ensure that you are not specifying
--security-protocol PLAINTEXTSASL
argument to the consumer. Either specify--security-protocol PLAINTEXT
or leave--security-protocol
unspecified since its default value isPLAINTEXT
.
I can’t edit the /etc/kafka/conf/kafka_client_jaas.conf
file! What should I do to consume kafka messages over an non-authenticated channel?
- In that case just do a
kinit
with a valid password/ticket. -
That token will get used to authenticate you to zookeeper. After that you should be able to consume messages from kafka over non-authenticated channel. Connection to Kafka brokers correctly happens over non-authenticated channel and should get authorized as user
ANONYMOUS
.
Why do I need to edit the /etc/kafka/conf/kafka_client_jaas.conf
file?
Presence of Client block in /etc/kafka/conf/kafka_client_jaas.conf
for service zookeeper
causes the console consumer connect to zookeeper in secure mode. To do so it needs a ticket -- which won’t exist in simple auth mode, so it fails.
Authorizing topic creation
This section describes the issues one might encounter while trying to authorize topic creation in Kafka using Ranger.
Can I authorizer topic creation via Ranger?
Yes, but only if the topic is being auto-created by consumers or producers.
What is the recommended policy setup to authorize topic auto-creation for producers or consumers?
- Create a policy where resource is all topics, i.e. *.
-
For producers, create a policy item under this policy which grants both
Produce
andConfigure
permissions to the relevant user/user-groups. -
For consumers, create a policy item under this policy which grants both
Consume
andConfigure
permissions to the relevant user/user-groups.
Can I authorize topic auto-creation for producers or consumers that connect over non-authenticated channel?
- Yes, create a policy similar to that for secure producer.
- Either add user group
public
to the policy item or specify andip-address
base custom condition. - Refer to FAQ about authorizing Kafka access over non-authenticated channel for additional details and rationale.
Why do I have to grant create access to all topics (via *) to allow for auto-creation to work for producers and/or consumers?
Topic creation is currently a cluster level privilege. Thus it requires access privileges over all topics in a cluster, i.e. *.
I want to allow topic auto creation for any topic that starts with finance
, e.g. finance_1
, finance_2
, etc. to users that are part of Finance
user group. But I don’t want them to be able to auto create topics that start with other strings, say, marketing_123
. Can I model this sort of an authorization in Ranger Kafka plugin?
- No. Because in Kafka currently topic creation is a cluster level permissions, i.e. all topics.
- There is a pending proposal about Hierarchical topics in Kafka which, if and when it’s implemented, could help with that use case.
I am using the Kafka supplied console consumer to test topic auto creation by a consumer, but it is not working. Shouldn’t the new topic get auto-created the moment I startup the consumer? I have verified the recommended policy setup as indicated above! What gives?
Make sure that you specify the following two argument to the console consumer.
--new-consumer
--boot-strap <broker-name(s)>
: Any single broker host/port would do.
Most common way of creating topic involves using the bin/kafka-tpics.sh
script that is a part of the Kafka distribution. Can I authorize topic creation via that mechanism?
No.
Why can’t I authorize topic creation done via the bin/kafka-tpics.sh script!?
- This script talks directly to zookeeper. Hence, the policies of Kafka plugin don’t come into the picture.
- Script adds entries into zookeeper nodes and watchers inside the brokers monitor it and create topics.
So what are my options to authorize topic creation via the bin/kafka-tpics.sh script?
- Since this directly interacts with zookeeper this is best controlled via zookeeper acls.
Is there a Ranger plugin for Zookeeper?
Not yet.
Where can I learn more about Kafka’s support for publish/consume over non-authenticated channel?
Please refer to KAFKA-1809 which implemented the multiple listeners Design.