Download Secured Artifactory Artifact With WGET and Token

If you need to download a secured Artifactory object from an automation server, and you don’t have a service account that you can use, you can go to artifactory for your user, generate a token, and get it.

Then you can simply do this:

wget --header='X-JFrog-Art-Api: your-very-long-token-from-artifactory' https://company.com/artifactory/local-pypi-repo/some_repo/some_project/artifact_name-3.1.0-py3-none-any.whl

WGet will use the artifactory token in its header and artifactory will allow you to download the artifact as if you are yourself.

Word of caution; while you haven’t revealed your user-name and password, this token can effectively be used for any Artifactory API as if its you. So, be cautious with who else can see this still :).

Installing Jenkins on Centos 7.x for Docker Image Builds

First, just start by getting a root shell so we can drop the sudo command from everything (e.g. type sudo bash).  This isn’t best practice, but just make sure you exit out of it when you are done :).

Centos 7.x Complete Installation Steps

Note that this installs wget, OpenJDK Java 1.8, Jenkins, enables Jenkins to be on with system start, and then it installs docker, and finally, it adds the Jenkins user to the docker user group so that it can run commands from it effectively (using elevated privileges).  Lastly, it restarts Jenkins so that the user has the new docker group permissions in the running process.

yum update -y
yum install wget -y
yum install java-1.8.0-openjdk-devel -y
wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
rpm –import https://jenkins-ci.org/redhat/jenkins-ci.org.key
yum install jenkins -y
service jenkins start
chkconfig jenkins on
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager –add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io
systemctl start docker
usermod -a -G docker jenkins
service jenkins restart
 

Validation

Assuming this all worked, you should see Jenkins running on your localhost:8080 port for that server, and you can follow its on-screen instructions.

For docker, you can run the hello world container to see if it is properly set up (docker run hello-world).

Resources

I got this information from the Jenkins wiki, the docker site itself, and one stack overflow entry as shown below:

Docker – Map IP to DNS, or Just Override DNS Name in Linux in General

I was reading a docker script someone else created and came across an interesting blog explaining a parameter it used (–add-host) right here.  I recommend reading that longer blog, but I’m recording the short notes and link here for myself as I’m sure I’ll be using this a lot.

The “–add-host” Parameter

Long story short, you can just add “–add-host=some_dns_name:some_ip_address” to your docker command in order to make your container have any DNS name resolve to any IP address.

This works by having the container put an entry for this DNS/IP pair into the /etc/hosts file.

Use Outside of Docker

I haven’t used the /etc/hosts file in a while.  But this reminded me about it.  The article points out that you can either add a DNS mapping or even override an existing DNS mapping using this file.

So, for example, I could make google.com point at this website from within the given OS instance, if I updated that file properly.

Pretty cool and useful :).

Presto Custom Password Authentication Plugin (Internal)

Presto Authentication (Out of the Box)

Out of the box, presto will not make you authenticate to run any queries.  For example, you can just connect with JDBC from Java or DBeaver/etc and run whatever queries you want with any user name and no password.

When you want to enable a password, it has a few options out of the box:

So, unfortunately, if you just want to create some users/passwords and hand them out, or get users from an existing system or database, there really isn’t a way to do it here.

Custom Password Authenticator

Most things in Presto are implemented as plugins, including many of its own out-of-the-box features.  This is true for the LDAP authenticator above.  It actually is a “Password Authenticator” plugin implementation.

So, if we copy the LDAP plugin and modify it, we can actually make our own password plugin that lets us use any user-name and password/etc we want!  Note that we’ll just make another “internal” plugin which means we have to recompile presto.  I’ll try to make this an external plugin in another article.

Let’s Do It!

Note: we will be modifying presto code, and presto code only builds on Linux.  I use windows, so I do my work on an Ubuntu desktop in the cloud; but you can do whatever you like.  If you have a Linux desktop, it should build very easily out of Intellij or externally with Maven.

  1. Clone the presto source code.
  2. Open it in Intellij or your editor of choice.
  3. Find the “presto-password-authenticators” sub-project and navigate to com.facebook.presto.password.
  4. Copy LdapAuthenticator.java to MyAuthenticator.java.
  5. Copy LdapConfig to MyAuthenticatorConfig.java.
  6. Copy LdapAuthenticatorFactory to MyAuthenticatorFactory.java.
  7. For MyAutnenticatorConfig, make a password and a user private variable, both strings.  Then make getters and setters for them similar to the LDAP URL ones; though you can take out the patterns/etc.  You can remove everything else; our user name and password will be our only config.
  8. For MyAuthenticatorFactory, change getName() to return “my-authenticator”.  Also edit it so that it uses your new config class and so that it returns your new authenticator in config.
  9. For MyAuthenticator, just make the private Principal authenticate(String user, String password) {} method throw AccessDeniedException if the user and password don’t match the ones from your config.  You can get the ones from your config in the constructor.
  10. Add MyAuthenticator to the PasswordAuthenticationPlugin.java file under the LdapAuthenticatorFactory instance.

This is all the code changes you need for your new authenticator (assuming it works).  But how do we use/test it?

  • The code runs from the presto-main project.
  • Go there and edit config.properties; add these properties:
http-server.authentication.type=PASSWORD
http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/etc/presto-keystore/keystore.jks
http-server.https.keystore.key=somePassword
  • Then add a password-authenticator.properties in the same directory with these properties:
password-authenticator.name=my-authenticator
user=myuser
password=otherPassword
  • Note that the password-authenticator.name is equal to the string you used in the factory class in step #8.
  • The first config file tells it to use a password plugin, the second one tells it which one to use based on that name and what extra properties to give to it based on the config (in our case, just a user and a password).

The Hard Part

Okay, so all of this was pretty easy to follow.  Unfortunately, if you go try to run this, it won’t work (Note that you can run the presto in IntelliJ by following these instructions).

Anyway, your plugin won’t work because password plugins don’t do anything unless you have HTTPS enabled on the coordinator.   This is because presto developers don’t want you sending clear-text passwords; so the password plugin type just makes it flat out not work!

We put properties in our first config file for HTTPS already.  Now you need to follow the JKS instructions here: https://prestosql.io/docs/current/security/tls.html#server-java-keystore to generate a key store and put it at the path from our config file.

Note that if you’re using AWS/etc, the “first and last name” should be the DNS name of your server.  If you put the IP, it won’t work.

Once you’ve got that set up, you can start presto and connect via JDBC from the local host by filling out the parameters noted here (https://prestosql.io/docs/current/installation/jdbc.html).

SSL Use HTTPS for connections
SSLKeyStorePath The location of the Java KeyStore file that contains the certificate and private key to use for authentication.
SSLKeyStorePassword The password for the KeyStore.

SSL = true, key store path is the one from the config file earlier, and the password is also the one from the config file.

Now, if you connect via JDBC and use the user name and password from your config file, everything should work!  Note that you probably want to get a real certificate in the future or else you’ll have to copy yours key store to each computer/node you use so JDBC works on each (which isn’t great).

UPDATE – 2019/08/04 – YOU should make clients use a trust store with the private key removed for numerous reasons.  See this newer post for how to take this JKS file and modify it for client use properly: https://coding-stream-of-consciousness.com/2019/08/04/presto-internal-tls-password-login-removing-private-key-from-jks-file/.

Hive SQL Standard Authorization – Not Column Level

Hive SQL Standard Authorization Setup

This confluence page https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization will walk you through SQL standard authorization in Hive.  The bottom of the page gives advice on setting it up (and it works well on hive 2.3.5 as I just tried it).

As of 6/15/2019, the set up config noted (tested on 2.3.5) is:

Set the following in hive-site.xml:

  • hive.server2.enable.doAs to false.
  • hive.users.in.admin.role to the list of comma-separated users who need to be added to admin role. Note that a user who belongs to the admin role needs to run the “set role” command before getting the privileges of the admin role, as this role is not in current roles by default.
  • Add org.apache.hadoop.hive.ql.security.authorization.MetaStoreAuthzAPIAuthorizerEmbedOnly to hive.security.metastore.authorization.manager. (It takes a comma separated list, so you can add it along with StorageBasedAuthorization parameter, if you want to enable that as well).
    This setting disallows any of the authorization api calls to be invoked in a remote metastore. HiveServer2 can be configured to use embedded metastore, and that will allow it to invoke metastore authorization api. Hive cli and any other remote metastore users would be denied authorization when they try to make authorization api calls. This restricts the authorization api to privileged HiveServer2 process. You should also ensure that the metastore rdbms access is restricted to the metastore server and hiverserver2.
  • hive.security.authorization.manager to org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory. This will ensure that any table or views created by hive-cli have default privileges granted for the owner.

Set the following in hiveserver2-site.xml:

  • hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory
  • hive.security.authorization.enabled=true
  • hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
  • hive.metastore.uris=’ ‘

This indeed works well.  After doing this, I was able to set the admin role for my noted user.  This let me create more roles, apply grants on them, and assign them to other users.  Users correctly were rejected from doing queries when they did not have the appropriate grants.

Note that I did not set up any kind of password authentication yet, I was just providing a user name to the hive JDBC connection with no password.  But the grants were properly respected based on the user name.

create role testrole;
grant role testrole to user testuser;
grant select on testdb.sample_data to role testrole;
show grant on table testdb.sample_data;
show current roles;
show principals testrole;

-- Fails for testuser before testdb.sample_data grant is given.
select * from testdb.sample_data;

No Column Level Grants/Restrictions

Unfortunately, I could not find any valid syntax for applying column level permissions, even though columns are shown in the grant outputs.  I also cannot find anything online to imply that permissions can be column level.  So, I think this feature is not supported.  You have to live with table level permissions.

That probably makes sense given there are 2 vendor products in the space for this (Apache Ranger and Apache Sentry).

Custom HiveAuthorizationProvider Example to Block Column Access

Hive has a very good set of hooks which you can use to customize all kinds of things.  It also has other “pluggable” areas which are basically hooks, but that aren’t called as such.

Here is a great article to get you started on Hive Hooks -> http://dharmeshkakadia.github.io/hive-hook/.

Creating a HiveAuthorizationProvider

In this case we aren’t implementing a hook specifically, but we’re doing the same exact flow to create our own HiveAuthorizationProvider.  We’ll do a very simple example to just block access to any column named “description” (as a silly example).

package com.john.humphreys.hive;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hive.metastore.api.Database;
import org.apache.hadoop.hive.ql.metadata.AuthorizationException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.metadata.Partition;
import org.apache.hadoop.hive.ql.metadata.Table;
import org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase;
import org.apache.hadoop.hive.ql.security.authorization.Privilege;

import java.util.List;

public class MyHiveAuthorizationProvider
        extends HiveAuthorizationProviderBase {
    @Override
    public void init(Configuration conf) throws HiveException {

    }

    @Override
    public void authorize(Privilege[] readRequiredPriv, Privilege[] writeRequiredPriv) throws HiveException, AuthorizationException {

    }

    @Override
    public void authorize(Database db, Privilege[] readRequiredPriv, Privilege[] writeRequiredPriv) throws HiveException, AuthorizationException {

    }

    @Override
    public void authorize(Table table, Privilege[] readRequiredPriv, Privilege[] writeRequiredPriv) throws HiveException, AuthorizationException {

    }

    @Override
    public void authorize(Partition part, Privilege[] readRequiredPriv, Privilege[] writeRequiredPriv) throws HiveException, AuthorizationException {

    }

    @Override
    public void authorize(Table table, Partition part, List<String> columns, Privilege[] readRequiredPriv, Privilege[] writeRequiredPriv) throws HiveException, AuthorizationException {
        if (columns.contains("description")) {
            throw new AuthorizationException("Not allowed to select description column!");
        }
    }
}

The only dependency required by this in maven is:

<dependency>
  <groupId>org.apache.hive</groupId>
  <artifactId>hive-exec</artifactId>
  <version>2.3.5</version>
</dependency>

You literally don’t need any special build plugins or anything. If you build a project with just this (Java 1.8), and you take the JAR file, and you put it in your hive/lib folder, then you’re almost ready.

The last step is to modify your hive-site.xml and to add these 2 properties:

<property>
  <name>hive.security.authorization.enabled</name>
  <value>true</value>
</property>

<property>
  <name>hive.security.authorization.manager</name>
  <value>com.john.humphreys.hive.MyHiveAuthorizationProvider</value>
</property>

After that, restart your hiveserver2, and when you try to select the “description” column from any table with it, it will get rejected.

Example In Practice

If I have a table called sample_data and I have a description column in it, and I run this query:

select * from (
    select * from (
        select description from sample_data
        ) x
    ) y;

I get this result:

Query execution failed

Reason:
SQL Error [403] [42000]: Error while compiling statement: Not allowed to select description column!

So, we can see it worked properly.

Limitations

Unfortunately, while this guards hive, it surprisingly doesn’t guard Presto when it access data via the hive metastore. So, as I need to guard hive and presto, I need to understand why and see if there is some other option.

Java – Get Parent Class Private Variable in Sub Class

I’m working on making a derived version of the Presto Hive plugin.  Unfortunately, the base class I need to inherit from uses its own private class loader, and the function I need to override (which is override-able!) for some reason requires that class loader as a parameter.

Anyway, long story short, I need to get the parent object’s private field to use it in the sub class I’m creating.  Reflection to the rescue!

Note: This is not generally good programming practice. Understand what the code does and why it does it before doing this.

Solution

//Class A file.

public class ClassA {
    private String name;
    public ClassA() {
        this.name = "Hello World!";
    }
}

// Class B file.

import java.lang.reflect.Field;

public class ClassB extends ClassA {
    public ClassB() {
        super();
    }

    public void printSuperPrivateMember() throws Exception {
        Field nameField = ClassA.class.getDeclaredField("name");
        nameField.setAccessible(true);
        System.out.println((String) nameField.get(this));
    }

    public static void main(String[] args) throws Exception {
        ClassB b = new ClassB();
        b.printSuperPrivateMember();
    }
}