Powershell – Disable Windows 10 Service on PC Unlock

My Issue

Recently I was running on a computer with a policy that was starting a (clearly) malfunctioning program on start-up which basically permanently was taking 100% of my CPU.

I didn’t have the ability to change the policy, but I did have the ability to stop the service.  Unfortunately, it regularly gets brought back as manually enabled and started.

The Fix

In cases like this, on Windows 10, you can search “Task Scheduler” on your start menu.  Then create a new task (not a basic task) that runs with highest privileges.  You can tell it to trigger on PC unlock, and have it start a program.

For the program, you can give your path to powershell (e.g. C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe).  Then you can give the parameters “-windowstyle hidden C:\dev\disable-service.ps1”.  Honestly, the window style hidden isn’t working but since it just runs on unlock its not a big deal; I’ll figure that out later.

The disable-service.ps1 script should have content like this:

Set-Service "Your Service Name" -StartupType Disabled
Stop-Service "Your Service Name"

You can get the full service name from your services utility in Windows. Note that the real service name needs to be used, and that may not be what you see specifically in the services utility or task manager. But if you click on the service in the services utility, you can get this real name easily.

Presto Hive External Table TEXTFILE Limitations

Background

I was hoping to use hive 2.x with just the hive metastore and not the hive server or hadoop (map-reduce).  Part of this plan was to be able to create tables within Presto; Facebook’s distributed query engine, which can operate over hive, in addition to many other things.

Initial Test Failure – CSV Delimited File

I’m still trying to decide if this is viable or not,  but my first test was not so great.

I created a new database and an external table pointing to a file on my AWS s3 bucket.  The file was a simple CSV delimited file.

Here was the “SQL” I used in Presto:

create schema testdb;

CREATE TABLE testdb.sample_data (
  x varchar(30), y varchar(30), sum varchar(30)
)
WITH (
  format = 'TEXTFILE',
  external_location = 's3a://uat-hive-warehouse/sample_data/'
);

use testdb;

select * from sample_data;

This all ran great, but unfortunately, my results looked like this:

x     | y    | sum
--------------------
1,2,3 | null | null
4,5,6 | null | null
...

The Problem

So, it turns out that Presto over hive, when targeting text files, does not support arbitrarily delimited files. In my case, I was pointing to a simple CSV file. So, it literally read all of the data into the first column and gave up.

It is actually viewing the file as delimited; but it only supports the \001 delimiter; ASCII control code 1, which is unprintable.

You can still go use parquet, orc, or whatever you want – but this makes CSV more than useless from Presto’s perspective. You can go create the table in hive first and it probably will work; but Presto cannot be the creator in this case :(.

Hive 3 Standalone Metastore + Presto

Hive 3.0 Standalone Metastore – Why?

Hive version 3.0 allows you to download a standalone metastore.  This is cool because it does not require you to deploy hadoop and/or run the rest of Hive’s fairly large deployment.  This makes a lot of sense because many tools that use hive for schema management do not actually care about Hive’s query engine.

For example, Presto is a clustered query engine in its own right; it has no interest in using hadoop/map-reduce to execute a query on hive data; it just wants to view and manage hive’s metadata through its thrift metastore interface.  Similarly, Apache Spark loves to work with hive, but it actually goes directly to the underlying database for performance reasons and works against that.  So, it also does not need hive’s query engine.

Can/Should We Use It?

Unfortunately, Presto only currently supports Hive 2.X.  From it’s own documentation: “The Hive connector supports Apache Hadoop 2.x and derivative distributions including Cloudera CDH 5 and Hortonworks Data Platform (HDP).”

If you read online though, you will find that it does seem to work… but with limited features.  If you look at this git entry for example: https://groups.google.com/forum/#!topic/presto-users/iAeEecsnS9I, you will see:

“We have tested Presto 0.203e with Hive 3.0 Metastore, and it works fine. We tested it by running TPC-DS queries, and Presto completed all 99 queries.”

But lower down, you will see:

However, Presto is not able to read Hive managed (transactional tables) in Hive 3.x…

Yes, this is a known limitation.

Unfortunately, transactional ACID v2 tables are the default for Hive 3.x.  So, basically all managed tables will not work in Hive 3.x even though external tables will work.  So, it might be okay to use it if you only do external tables… but in our case we let people use Spark however they like and they likely create many managed tables.  So, this rules out using Hive 3.0 with the standalone metastore for us.

I’m going to see if Hive 2.0 can be run without the hive server and hadoop next.

Site Note – SchemaTool

I would just like to make a side-note that while I did manage to run the Hive Standalone Metastore without installing hadoop, I did have to install (but not run) hadoop in order to use the schematool provided with hive for creating the hive RDMBS schema.  This is due to library dependencies.

There is a “create on first run” config you can do instead of this as well but they don’t recommend using it in production; so just keep that in mind.

Useful Links

Ansible Vault – Encrypt Single Strings in Config Files (Not Whole File)

Why Use Ansible Vault?

Ansible vault is a very useful tool in the world of devops.  It provides you an easy way to check your entire configuration into version control (like git) without actually revealing your passwords.

This means that when you apply config changes, it gets recorded and you can roll-back or upgrade just like you do with normal code.  Nothing is lost, and nothing is insecure.

Encrypting a Single Property

Ansible can be used in countless ways; it is a very flexible tool.  Take variables for example; you can put them in your inventory file, in separate variable files, at the top of your playbook, in the CLI command, and I’m guessing even more places.

In this case, lets say you have a nice existing variables file that just happens to not be encrypted.  You don’t want to split up your file and you don’t want to encrypt the whole file either because then it is hard for others to review (e.g. why encrypt the DNS name for a service you’re hitting?).

Example vars-dev.yml:

mysql_password: password123
dns_name: some-service@company.com

So, let’s encrypt just the password in that file and leave everything else as plain text. We’ll use this command to encrypt the password (password123).

$> ansible-vault encrypt_string --vault-id vault_password 'password123' --name 'mysql_password'
mysql_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
31336265626432653930373332313963666565336333303731653133613732616234383634613134
6463393739616232613837363065376336346265646432360a383464326231333035633762663738
65396562386166613537346165646539303334383039396235636331343830633761303239643434
6335343035646665640a666338346130636161663561376662666566316565646133653162663065
3534
Encryption successful

Note that “vault_password” in this line is actually a text file in the current directory that just has the password to my vault in it. Keeping your vault password in a file lets you use it in automation like Jenkins without it showing up in log entries/prompts.

$> cat vault_password
some_good_password

The entire part from ‘mysql_password’ onward in the command output (prior to Encryption Successful) is the line you should now use for the password in your variables file. So, the modified variables file looks like this:

Updated Example vars-dev.yml:

mysql_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
31336265626432653930373332313963666565336333303731653133613732616234383634613134
6463393739616232613837363065376336346265646432360a383464326231333035633762663738
65396562386166613537346165646539303334383039396235636331343830633761303239643434
6335343035646665640a666338346130636161663561376662666566316565646133653162663065
3534

dns_name: some-service@company.com

Testing It Out

At this point, let’s test that we can use this partially encrypted variable file for something real.

Let’s start by creating some new files, all of which we’ll be lazy with and just keep in our current directory:

We’ll create a config.properties template file that uses our variables.

mysql.password={{mysql_password}}
dns.name={{dns_name}}

Well create a playbook.yml file that is our actual playbook to run:

---

 - name: Ansible Vault Example
   hosts: my_hosts
   tasks:
     - name: Copy config file to servers with decrypted password.
       template: src="config.properties" dest="/tmp/config.properties"

And we’ll create inventory.yml to tell ourselves which hosts to target. We’ll just target localhost, but whatever. Make sure localhost can ssh to itself for this :).

[my_hosts]
localhost

Now we can run our test! Use this command to run our playbook with our inventory file and our variables file with our vault password. Based on the playbook, it should copy config.properties from our current directory to /tmp/config.properties with the variables replaced and decrypted where relevant.

ansible-playbook -i inventory.yml --vault-password-file vault_password --extra-vars @vars-dev.yml playbook.yml

Now we can cat /tmp/config.properties and see that it worked!

$> cat /tmp/config.properties
mysql.password=password123
dns.name=some-service@company.com

I hope this helps you in the future! I’ll be doing more blogs on Ansible, Vault, etc in the near future. So, come back for more!

Jenkins Pipeline Maven /Build + Deploy to Nexus/Artifactory in Docker

Overview

If you want to use a Jenkins pipeline to build your maven project in a clean docker container, you can do so as shown below.

You can do any maven commands you like.  In this particular case I am:

  • Setting the version in the POM to the one provided by the person running the Jenkins job.
  • Building and deploying the maven project to Artifactory.

This is the “build one” of “build once deploy everywhere”.

To test this though, you can just swap my commands out for maven –version or something simple like that.

Configuration

Create  a new pipeline job and:

  1. Add a string parameter called VERSION_TO_TAG.
  2. Set pipeline definition to “Pipeline script from SCM”.
  3. Give your repo checkout URL – e.g. ssh://git@url.mycompany.com:7999/PROJECT/repo-name.git.
  4. Specify your git credentials to use.
  5. Specify your Jenkinsfile path (It is probably literally just “Jenkinsfile” which is the default if you have the Jenkinsfile in the root of your repo).

Make sure your project has a “Jenkinsfile” at its root with a definition like this:

pipeline {
    agent {
        docker { image 'maven:3-alpine' }
    }

    stages {
        stage('Set version, build, and deploy to artifactory.') {
            steps {
                sh 'mvn versions:set -DnewVersion=$VERSION_TO_TAG'
                sh 'mvn deploy'
            }
        }
    }
}

Now, when you build your project, you should see it docker-pull a maven:3-alpine image, start the container, and run our maven commands and upload artifacts to the Artifactory repository you have set up in your maven POM (in your distributionManagement section in case you haven’t done that yet).

Azure LB Dropping Traffic Mysteriously – HaProxy / NGNIX / Apache / etc.

Failure Overview

I lost a good portion of last week fighting dropping traffic / intermittent connection issues in a basic tier azure load balancer.  The project this was working on had been up and running for 6 months without configuration changes and had not been restarted in 100 days.  Restarting it did not help, so clearly something had changed about the environment.  It also started happening in multiple deployments in different azure subscriptions, implying that it was not an isolated issue or server/etc related.

Solution

After doing a crazy amount of tests and eventually escalating to Azure support, who reviewed the problem for over 12 hours, Azure support pointed out this:

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview#types

“Do not translate or proxy a health probe through the instance that receives the health probe to another instance in your VNet as this configuration can lead to cascading failures in your scenario. Consider the following scenario: a set of third-party appliances is deployed in the backend pool of a Load Balancer resource to provide scale and redundancy for the appliances and the health probe is configured to probe a port that the third-party appliance proxies or translates to other virtual machines behind the appliance. If you probe the same port you are using to translate or proxy requests to the other virtual machines behind the appliance, any probe response from a single virtual machine behind the appliance will mark the appliance itself dead. This configuration can lead to a cascading failure of the entire application scenario as a result of a single backend instance behind the appliance. The trigger can be an intermittent probe failure that will cause Load Balancer to mark down the original destination (the appliance instance) and in turn can disable your entire application scenario. Probe the health of the appliance itself instead.”

I was using a load balancer over a scale set, and the load balancer pointed at HaProxy, which was designed to route traffic to the “primary” server.  So, I wanted Azure’s load balancer to consider every server up as long as it could route to the “primary” server, even if other things on this server specifically were down.

But having the health probe check HAProxy meant that the health probe was routed to the “primary” server and triggered this error.

This seems like an Azure quirk to me… but they have it documented.  Once I switched the health probe to target something not routed by HaProxy the LB stabilized and everything was ok.

 

Spring Time out REST HTTP Calls With RestTemplate

No Timeouts By Default!

Spring’s RestTemplate is an extremely convenient way to make REST calls to web services.  But most people don’t realize initially that these calls have no timeout by default.  This means no connection timeout and no data call timeout.  So, potentially, your app can make a call that should take 1 second and could freeze up for a very long time if the back end is behaving badly.

Setting a Timeout

There are a lot of ways of doing this, but the best one I’ve seen recently (from this stackoverflow post) is to create the RestTemplate in an @Configuration class and then inject it into your services.  That way you know the RestTemplate you are using everywhere was configured properly with your desired timeouts.

Here is a full example.

package com.company.cloudops.config;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;
import java.time.Duration;

@Configuration
public class AppConfig {

    @Value("${rest.template.timeout}") private int restTemplateTimeoutMs;

    @Bean
    public RestTemplate restTemplate(RestTemplateBuilder builder) {
        return builder
                .setConnectTimeout(Duration.ofMillis(restTemplateTimeoutMs))
                .setReadTimeout(Duration.ofMillis(restTemplateTimeoutMs))
                .build();
    }
}

To use this RestTemplate in another Spring bean class, just pull it in with:

@Autowired private RestTemplate template;