Utils

Jupyter

November 20, 2023
Utils
Jupyter, Python

Memory Usage #

def memory():
    with open('/proc/meminfo', 'r') as mem:
        ret = {}
        tmp = 0
        for i in mem:
            sline = i.split()
            if str(sline[0]) == 'MemTotal:':
                ret['total'] = int(sline[1])
            elif str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'):
                tmp += int(sline[1])
        ret['free'] = tmp
        ret['used'] = int(ret['total']) - int(ret['free'])
    return ret

No Hang Up #

nohup jupyter notebook --no-browser > notebook.log 2>&1 &

Workaround: no cells output #

se = time.time()
print(train.rdd.getNumPartitions())
print(test.rdd.getNumPartitions())
e = time.time()
print("Training time = {}".format(e - se))

your_float_variable = (e - se)
comment = "Training time for getnumpartition:"

# Open the file in append mode and write the comment and variable
with open('output.txt', 'a') as f:
    f.write(f"{comment} {your_float_variable}\n")

VS Code Configuration & Set-up

November 17, 2023
Utils, Tutorials
Git, Ssh

Configuration #

Remote SSH #

Host machine
    Hostname machine.com
    User user_name
    IdentityFile path/to/ssh/key

Remote SSH - SSH Tunnel #

Host tunnel_machine
    Hostname machine.com
    User user_name
    IdentityFile path/to/ssh/key

Host machine_after_tunnel
    Hostname machine_after_tunnel.com
    User user_name
    IdentityFile path/to/ssh/key
    ForwardAgent yes
    ProxyJump tunnel_machine

PC Configuration #

Authorize your windows local machine to connect to remote machine.

$USER_AT_HOST="your-user-name-on-host@hostname"
$PUBKEYPATH="$HOME\.ssh\id_ed25519.pub"

$pubKey=(Get-Content "$PUBKEYPATH" | Out-String); ssh "$USER_AT_HOST" "mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '${pubKey}' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Verify that the authorized_keys file in the .ssh folder for your remote user on the SSH host is owned by you and no other user has permission to access it.

...

Run plotly in JupyterLab

October 24, 2023
Utils, Tutorials
Jupyter, Python-Librairies

    1  pip uninstall plotly
    2  jupyter labextension uninstall @jupyterlab/plotly-extension
    3  jupyter labextension uninstall jupyterlab-plotly 
    4  jupyter labextension uninstall plotlywidget
    5  jupyter labextension update --all
    6  pip install plotly==5.17.0
    7  pip install "jupyterlab>=3" "ipywidgets>=7.6"
    8  pip install jupyter-dash
    9  jupyter labextension list

Install python packages offline

June 20, 2023
Utils, Tutorials
Pip, Python

1- Download packages locally using a requirements file or download a single package

pip download -r requirements.txt
## Example - single package
python -m pip download \
--only-binary=:all: \
--platform manylinux1_x86_64 --platform linux_x86_64 --platform any \
--python-version 39 \
--implementation cp \
--abi cp39m --abi cp39 --abi abi3 --abi none \
scipy

2- Copy them to the a temporary folder in your remote machine 3- On your machine, Activate conda and then install them using pip - specify installation options

...

Hive

April 2, 2018
Utils
Sql, Hql, Development

Snippets #

-- set identifiers to none for the query below to work and 
-- set it back to column once it's done
set hive.support.quoted.identifiers = none;

HIVE 3 #

  • BI Code typically use db.table - needs to change to db.table
  • Default path : /warehouse/tablespace/external/hive/default.db/test_table

ACID + HIVE