Apache Airflow Windows 10 Install (Ubuntu)

After my failed attempt at installing Aifrflow into python on Windows the normal way, I heard that it is better to run it in an Ubuntu sub-system available in the Windows 10 store.  So, I’m changing to this route.

You can find and install “Ubuntu” on the Windows 10 store, and it will give you a full fledged Ubuntu Linux shell.  Here’s what the installation looks like:

Ubuntu Installation

It installs quite quickly, then you just press “Launch”.  The shell opens, and in my case, I was presented with this:

Installing, this may take a few minutes…
WslRegisterDistribution failed with error: 0x8007019e
The Windows Subsystem for Linux optional component is not enabled. Please enable it and try again.
See https://aka.ms/wslinstall for details.
Press any key to continue…

Go to your start menu and type “features” and click “Turn Windows features on or off”, then check the “Windows Subsystem for Linux” box and press “OK”.

It will install some things and take a few minutes.  For me, it took about 2 minutes on “Searching for required files” even though I’m on a very fast corporate internet connection.  So, don’t be discouraged if that happens.

Unfortunately, you’ll have to reboot once this finishes!  Such is windows :(.

After the reboot, open the “Ubuntu” shell from your windows button search, and then it will take a minute to install and will ask you to create a user and ID (note that “admin” will not work, so don’t bother trying that).

Installing, this may take a few minutes…
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username:
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Installation successful!
To run a command as administrator (user “root”), use “sudo “.
See “man sudo_root” for details.

If you check, you’ll already have python installed.  It is version 3.6.5 for me which is good, because a previous post where I tried to install it on windows showed that Airflow is not compatible (yet) with Python 3.7 when pip installing as it added the “async” keyword which broke some things.

$ python3 –version
Python 3.6.5

Now, we should just have to install Airflow.  But we need pip first, and when I try to install pip the way it recommends (when you try to use it as is), then it doesn’t work.  So, I found this: https://askubuntu.com/questions/672808/sudo-apt-get-install-python-pip-is-failing which recommends:

sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get update

After you run those commands, you can run the last one:

sudo apt-get install python-pip

This is actually the one the Ubuntu terminal recommended if you just tried to blindly use pip in the first place; but it wouldn’t have worked without the other 3 first.  This took around 5 minutes to install for me, and and it will require you to say “y” for yes once to kick it off.

After this, we can FINALLY install Airflow properly.  This is a pretty big victory if you realize that I started on my other blog post trying to make it work in Windows first, and that was a rabbit hole in itself!

export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install apache-airflow

If you’re wondering why that first export line is there, just skip it and read the terminal error message which recommends it.  I ran into the same thing in the pure Windows install which failed in the other blog post.

This installation took around 3 minutes for me.  The Airflow documentation recommends initializing its database (SQLite by default) when you’re done as other things won’t work without it – https://airflow.apache.org/installation.html:

Surprisingly, I found I had to open a new terminal before I could use the airflow command.  I’m not sure if this is a quirk about running it on windows, or if I should have just sourced my profile again/etc as I didn’t play around with it.

In any case, initialize the DB and then check the version, and hopefully you’re as happy as I am to be done with that.
 

airflow initdb

hujo8003@USLJ96YRQ2:~$ airflow version
[2018-11-06 11:36:38,930] {__init__.py:51} INFO - Using executor SequentialExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
v1.10.0

 

 

3 thoughts on “Apache Airflow Windows 10 Install (Ubuntu)

  1. Hi After Installing and following above steps i get error while using “airflow initdb”
    error : /mnt/c/ProgramData/Anaconda3/Scriprts/airflow : c:\…\..\python.exe^M: bad interpreter: No such file or directory

    i Tried with new terminal also
    You have any solution
    Thanks

    Like

    • You sure you followed this from a clean state? I don’t think Anaconda should even be present. Airflow just isn’t finding Python (which it relies on).

      Make sure python is on your path (which python) so airflow can find it.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s