Linux Neatly Manage SSH Sessions Like MRemoteNG on Windows

SSH technically works the same anywhere, but on Windows without special tooling, using SSH can be cumbersome as the CLI is not very… elegant?  Generally, on Windows, people use tools like putty or MRemoteNG (also putty based) to make things neater.

If you’re on Linux, you can use SSH out of the box very cleanly by leveraging the config file to create named hosts.

For example, this verbose command:

ssh -i ~/.ssh/prod-key.pem centos@10.20.30.40

Can be replaced by the following command – and the nice “pgadmin-prod” name even auto-completes.  So, if you have 100 servers for different things, you can still find them easily.

ssh pgadmin-prod

This works if you add the following block into your ~/.ssh/config file.

Host pgadmin-prod
User centos
HostName 10.20.30.40
IdentityFile ~/.ssh/prod-key.pem

Where:

  • Host = the user-friendly name I want to use for this connection.
  • User = the user I normally have to log in as.
  • HostName = the IP address of the target host.
  • IdentityFile = the path to the private key used for the connection.

You can have as many named hosts as you like and, again, their names will auto complete after the ssh command – so they’re very easy to use.

This gets even more effective if you combine it with a screen/session management app like tmux or screen.  So, I recommend you look into those too if you don’t use them already (tmux is a little more modern if you’re a new to both).

I hope this helps you be a little more efficient, and thanks for reading.

Docker – Map IP to DNS, or Just Override DNS Name in Linux in General

I was reading a docker script someone else created and came across an interesting blog explaining a parameter it used (–add-host) right here.  I recommend reading that longer blog, but I’m recording the short notes and link here for myself as I’m sure I’ll be using this a lot.

The “–add-host” Parameter

Long story short, you can just add “–add-host=some_dns_name:some_ip_address” to your docker command in order to make your container have any DNS name resolve to any IP address.

This works by having the container put an entry for this DNS/IP pair into the /etc/hosts file.

Use Outside of Docker

I haven’t used the /etc/hosts file in a while.  But this reminded me about it.  The article points out that you can either add a DNS mapping or even override an existing DNS mapping using this file.

So, for example, I could make google.com point at this website from within the given OS instance, if I updated that file properly.

Pretty cool and useful :).

Bash – Grep (or Run Other Command) Only On Files Created This Week, Day.

Use Case

I just ran into a simple problem where I had to grep files on a server, but the directory had TONS and TONS of files in it.  I just wanted to target files created within the last week or so.

Working Command

It turns out this find command is very handy for this occasion.  It was taken and lightly modified from this unix stack-exchange post after a fair bit of searching.

find . -mtime -7 -exec grep "my_search_string" {} \;

Basically, it finds everything in “.” (the current directory) that was created in the last 7 days (as in 24 hour days, not from-this-morning days), and it executes the grep expression on it.

You can modify the timing however you want with mtime as well as change the target directory or command to execute, and of course you can pipe the output to whatever you want :).

Apache Airflow Windows 10 Install (Ubuntu)

After my failed attempt at installing Aifrflow into python on Windows the normal way, I heard that it is better to run it in an Ubuntu sub-system available in the Windows 10 store.  So, I’m changing to this route.

You can find and install “Ubuntu” on the Windows 10 store, and it will give you a full fledged Ubuntu Linux shell.  Here’s what the installation looks like:

Ubuntu Installation

It installs quite quickly, then you just press “Launch”.  The shell opens, and in my case, I was presented with this:

Installing, this may take a few minutes…
WslRegisterDistribution failed with error: 0x8007019e
The Windows Subsystem for Linux optional component is not enabled. Please enable it and try again.
See https://aka.ms/wslinstall for details.
Press any key to continue…

Go to your start menu and type “features” and click “Turn Windows features on or off”, then check the “Windows Subsystem for Linux” box and press “OK”.

It will install some things and take a few minutes.  For me, it took about 2 minutes on “Searching for required files” even though I’m on a very fast corporate internet connection.  So, don’t be discouraged if that happens.

Unfortunately, you’ll have to reboot once this finishes!  Such is windows :(.

After the reboot, open the “Ubuntu” shell from your windows button search, and then it will take a minute to install and will ask you to create a user and ID (note that “admin” will not work, so don’t bother trying that).

Installing, this may take a few minutes…
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username:
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Installation successful!
To run a command as administrator (user “root”), use “sudo “.
See “man sudo_root” for details.

If you check, you’ll already have python installed.  It is version 3.6.5 for me which is good, because a previous post where I tried to install it on windows showed that Airflow is not compatible (yet) with Python 3.7 when pip installing as it added the “async” keyword which broke some things.

$ python3 –version
Python 3.6.5

Now, we should just have to install Airflow.  But we need pip first, and when I try to install pip the way it recommends (when you try to use it as is), then it doesn’t work.  So, I found this: https://askubuntu.com/questions/672808/sudo-apt-get-install-python-pip-is-failing which recommends:

sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get update

After you run those commands, you can run the last one:

sudo apt-get install python-pip

This is actually the one the Ubuntu terminal recommended if you just tried to blindly use pip in the first place; but it wouldn’t have worked without the other 3 first.  This took around 5 minutes to install for me, and and it will require you to say “y” for yes once to kick it off.

After this, we can FINALLY install Airflow properly.  This is a pretty big victory if you realize that I started on my other blog post trying to make it work in Windows first, and that was a rabbit hole in itself!

export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install apache-airflow

If you’re wondering why that first export line is there, just skip it and read the terminal error message which recommends it.  I ran into the same thing in the pure Windows install which failed in the other blog post.

This installation took around 3 minutes for me.  The Airflow documentation recommends initializing its database (SQLite by default) when you’re done as other things won’t work without it – https://airflow.apache.org/installation.html:

Surprisingly, I found I had to open a new terminal before I could use the airflow command.  I’m not sure if this is a quirk about running it on windows, or if I should have just sourced my profile again/etc as I didn’t play around with it.

In any case, initialize the DB and then check the version, and hopefully you’re as happy as I am to be done with that.
 

airflow initdb

hujo8003@USLJ96YRQ2:~$ airflow version
[2018-11-06 11:36:38,930] {__init__.py:51} INFO - Using executor SequentialExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
v1.10.0