Installing Python3 from Source and Set Up Remote Jupyter Notebook

This text explains how to install the newest python version and set up jupyter on a machine so it can be used remotely in Ubuntu. I tested it for version 14.04 but it should work the same for all Linux distributions.

Install python3 from source

Some prerequisite packages.

sudo apt-get install libssl-dev openssl libjpeg-dev libsqlite3-dev libbz2-dev

Now to compile and install Python

# Create install dir
mkdir Python3.5.2
# Get source
wget https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz
tar -zxvf Python-3.5.2.tgz
cd Python-3.5.2
# Build
./configure --prefix=../Python3.5.2/ --enable-shared
make -j12 # On 12 core machine
# Install
make install

Set up virtualenv

# Create venv
Python3.5.2/bin/pyvenv py3env
# Add shared library path to activate script
{ echo "export LD_LIBRARY_PATH=`pwd`/Python3.5.2/lib:\$LD_LOAD_PATH"; cat py3env/bin/activate; } > py3env/bin/activate.new
mv py3env/bin/activate.new py3env/bin/activate
# Activate venv
source py3env/bin/activate
# Update pip
pip install --upgrade pip
# Install additional packages
pip install numpy scipy matplotlib seaborn keras scikit-image scikit-learn jupyter

Set up Jupyter

Set up

We want Jupyter to be reachable over the network with a login.

# Create new jupyter config (answer yes on overwrite)
jupyter notebook --generate-config

Securing

Start a python interpreter and type

from notebook.auth import passwd
passwd()
# Enter password:
# Verify password:
# Out[2]: 'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

Save the returned string. Open .jupyter/jupyter_notebook_config.py and add/change the following.

c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

Warning: We are not setting up SSL so the hashed password will be transferred unencrypted. If you run jupyter on a server reachable on the internet it is probably a good idea to also activate SSL encryption (see here).

Making it reachable

Uncomment/Change the following in jupyter_notebook_config.py

c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False

# It is a good idea to set a known, fixed port for server access
c.NotebookApp.port = 9999

The server can then be started using jupyter notebook and be reached at ip:9999 in a browser.

Running Jupyter with caffe

Prerequisites

We have to install some prerequisites

pip install scikit-image protobuf pillow

Get Python to find it

For caffe to be available in the just installed jupyter we have to add the python caffe dir to PYTHONPATH. Add the following line to py3env/bin/activate

export PYTHONPATH=~/git/caffe/python/:$PYTHONPATH

Warning: Caffe currently only works with python2.7 so to use it you have to do the same as above but with python2.7

Get Keras to run in Jupyter on GPU using cuDNN

Edit $HOME/.keras/keras.json

{
    "floatx": "float32",
    "backend": "theano",
    "image_dim_ordering": "th",
    "epsilon": 1e-07
}

Add the following lines to py3env/bin/activate (Change the PATH to the actual CUDA path)

export PATH=$PATH:/usr/local/cuda/bin
export THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32

Using cuDNN for speedup

Download from official site, unpack, then copy the files to the CUDA install directories:

cd cuda
sudo cp lib64/* /usr/local/cuda/lib64/
sudo cp include/* /usr/local/cuda/include/

Author: Sebastian Stabinger

Email: firstname@lastname.name

Created: 2018-01-01 Mon 20:07

Emacs 25.3.1 (Org mode 8.2.10)

Validate