Tag Archives: install

Apache Airflow Windows 10 Install (Ubuntu)

Posted on by
12

After my failed attempt at installing Aifrflow into python on Windows the normal way, I heard that it is better to run it in an Ubuntu sub-system available in the Windows 10 store.  So, I’m changing to this route.

You can find and install “Ubuntu” on the Windows 10 store, and it will give you a full fledged Ubuntu Linux shell.  Here’s what the installation looks like:

Ubuntu Installation

It installs quite quickly, then you just press “Launch”.  The shell opens, and in my case, I was presented with this:

Installing, this may take a few minutes…
WslRegisterDistribution failed with error: 0x8007019e
The Windows Subsystem for Linux optional component is not enabled. Please enable it and try again.
See https://aka.ms/wslinstall for details.
Press any key to continue…

Go to your start menu and type “features” and click “Turn Windows features on or off”, then check the “Windows Subsystem for Linux” box and press “OK”.

It will install some things and take a few minutes.  For me, it took about 2 minutes on “Searching for required files” even though I’m on a very fast corporate internet connection.  So, don’t be discouraged if that happens.

Unfortunately, you’ll have to reboot once this finishes!  Such is windows :(.

After the reboot, open the “Ubuntu” shell from your windows button search, and then it will take a minute to install and will ask you to create a user and ID (note that “admin” will not work, so don’t bother trying that).

Installing, this may take a few minutes…
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username:
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Installation successful!
To run a command as administrator (user “root”), use “sudo “.
See “man sudo_root” for details.

If you check, you’ll already have python installed.  It is version 3.6.5 for me which is good, because a previous post where I tried to install it on windows showed that Airflow is not compatible (yet) with Python 3.7 when pip installing as it added the “async” keyword which broke some things.

$ python3 –version
Python 3.6.5

Now, we should just have to install Airflow.  But we need pip first, and when I try to install pip the way it recommends (when you try to use it as is), then it doesn’t work.  So, I found this: https://askubuntu.com/questions/672808/sudo-apt-get-install-python-pip-is-failing which recommends:

sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get update

After you run those commands, you can run the last one:

sudo apt-get install python-pip

This is actually the one the Ubuntu terminal recommended if you just tried to blindly use pip in the first place; but it wouldn’t have worked without the other 3 first.  This took around 5 minutes to install for me, and and it will require you to say “y” for yes once to kick it off.

After this, we can FINALLY install Airflow properly.  This is a pretty big victory if you realize that I started on my other blog post trying to make it work in Windows first, and that was a rabbit hole in itself!

export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install apache-airflow

If you’re wondering why that first export line is there, just skip it and read the terminal error message which recommends it.  I ran into the same thing in the pure Windows install which failed in the other blog post.

This installation took around 3 minutes for me.  The Airflow documentation recommends initializing its database (SQLite by default) when you’re done as other things won’t work without it – https://airflow.apache.org/installation.html:

Surprisingly, I found I had to open a new terminal before I could use the airflow command.  I’m not sure if this is a quirk about running it on windows, or if I should have just sourced my profile again/etc as I didn’t play around with it.

In any case, initialize the DB and then check the version, and hopefully you’re as happy as I am to be done with that.
 

airflow initdb

hujo8003@USLJ96YRQ2:~$ airflow version
[2018-11-06 11:36:38,930] {__init__.py:51} INFO - Using executor SequentialExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
v1.10.0

 

 

(Failed) Apache Airflow Windows-10 Install

Posted on by
1

NOTE: I fought this installation a lot and fixed numerous issues, but in the end I got stuck on a C++ compilation failure in the Airflow install via pip.  So, I’m switching to installing it in a new post in an Ubuntu shell available in the Windows 10 store since I’ll be running Airflow in Linux in production anyway.  So, there is probably some helpful stuff here; but its not a full solution and I recommend against doing Airflow this way given my experiences here.

I’m relatively new to Python and have only really used it for simplistic scripts in the past; but now I’ll be using it for a new job along with Apache Airflow (which is very cool).

Anyway, I just had a terrible time installing Airflow… so I thought I’d document the issues here and a working solution on Windows 10:

  1. Install python 3.6.7 from here: https://www.python.org/downloads/release/python-367/
    • (Do not use Python 3.7; as of 2018-11-06, “pip install apache-airflow” will install apache-airflow-1.10.0, and the installer will try and use the “async” keyword, which is now a reserved word in Python 3.7, so it will fail).
  2. Make sure Python and its Scripts directory are in your path (Python’s installer may or may not do this.  If you open a new command line after the Python install and “python –version” doesn’t show 3.6.7, you need to do it.
    • Note that the scripts directory is where pip is; this is your package installer for adding modules to Python.
  3. Upgrade pip with:  python -m pip install –upgrade pip
  4. The installation command for Airflow is “pip install apache-airflow”.  But in my case, this failed a few more times due to other dependencies/issues.  So, I had to do the following before this worked:
    • Set this environment variable: “set SLUGIFY_USES_TEXT_UNIDECODE=yes”
    • Install Microsoft Visual C++ 14 build packages (this is time consuming) and upgrade the build tools in Pip.
      • pip install –upgrade setuptools
      • install the “Build Tools for Visual Studio 2017” from: https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017
        • Once the interface opens for the installer,  install Visual C++ build tools – Build Windows desktop applications using the Microsoft C++ toolset, ATL, or MFC. I also checked the following boxes on the right:
          • Windows 10 SDK (10.0.17134.0)
          • Visual C++ tools for CMake
          • Testing tools core features – Build Tools
          • VC++ 2015.3 v14.00 (v140) toolset for desktop
          • Windows Universal CRT SDK
          • Windows 8.1 SDK
          • Installation is 2.5GB! Woah.
  5. Open a new command line so it picks up everything and then run
    • set SLUGIFY_USES_TEXT_UNIDECODE=yes
    • pip install apache-airflow
    • IT STILL DIDN’T WORK! – The error is a complex SYSTEM_THREADING related one and online docs for it seem to not have a resolution.  
      There’s probably a way to fix this; but at this point I’m going to switch to installing it in a Ubuntu shell subsystem from the Windows 10 store in a new blog post – I’ve wasted enough time on this given I’ll be running Airflow in Linux in production anyway.