November 29, 2023
BibTex
#
Limit number of authors in IEEEtran
#
In the .bib file configure your IEEEtran as follows:
@IEEEtranBSTCTL{IEEEexample:BSTcontrol,
CTLuse_forced_etal = "yes",
CTLmax_names_forced_etal = "3",
CTLnames_show_etal = "2" }
Cheat-sheets
#
November 20, 2023
Memory Usage
#
def memory():
with open('/proc/meminfo', 'r') as mem:
ret = {}
tmp = 0
for i in mem:
sline = i.split()
if str(sline[0]) == 'MemTotal:':
ret['total'] = int(sline[1])
elif str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'):
tmp += int(sline[1])
ret['free'] = tmp
ret['used'] = int(ret['total']) - int(ret['free'])
return ret
No Hang Up
#
nohup jupyter notebook --no-browser > notebook.log 2>&1 &
Workaround: no cells output
#
se = time.time()
print(train.rdd.getNumPartitions())
print(test.rdd.getNumPartitions())
e = time.time()
print("Training time = {}".format(e - se))
your_float_variable = (e - se)
comment = "Training time for getnumpartition:"
# Open the file in append mode and write the comment and variable
with open('output.txt', 'a') as f:
f.write(f"{comment} {your_float_variable}\n")
November 17, 2023
Configuration
#
Remote SSH
#
Host machine
Hostname machine.com
User user_name
IdentityFile path/to/ssh/key
Remote SSH - SSH Tunnel
#
Host tunnel_machine
Hostname machine.com
User user_name
IdentityFile path/to/ssh/key
Host machine_after_tunnel
Hostname machine_after_tunnel.com
User user_name
IdentityFile path/to/ssh/key
ForwardAgent yes
ProxyJump tunnel_machine
PC Configuration
#
Authorize your windows local machine to connect to remote machine.
$USER_AT_HOST="your-user-name-on-host@hostname"
$PUBKEYPATH="$HOME\.ssh\id_ed25519.pub"
$pubKey=(Get-Content "$PUBKEYPATH" | Out-String); ssh "$USER_AT_HOST" "mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '${pubKey}' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
Verify that the authorized_keys file in the .ssh folder for your remote user on the SSH host is owned by you and no other user has permission to access it.
...October 24, 2023
1 pip uninstall plotly
2 jupyter labextension uninstall @jupyterlab/plotly-extension
3 jupyter labextension uninstall jupyterlab-plotly
4 jupyter labextension uninstall plotlywidget
5 jupyter labextension update --all
6 pip install plotly==5.17.0
7 pip install "jupyterlab>=3" "ipywidgets>=7.6"
8 pip install jupyter-dash
9 jupyter labextension list
Useful Links
#
June 20, 2023
1- Download packages locally using a requirements file or download a single package
pip download -r requirements.txt
## Example - single package
python -m pip download \
--only-binary=:all: \
--platform manylinux1_x86_64 --platform linux_x86_64 --platform any \
--python-version 39 \
--implementation cp \
--abi cp39m --abi cp39 --abi abi3 --abi none \
scipy
2- Copy them to the a temporary folder in your remote machine
3- On your machine, Activate conda and then install them using pip - specify installation options
...January 20, 2020
Add
#
# only add files with .scala extension
git ls-files [path] | grep '\.scala$' | xargs git add
git stash --keep-index
April 2, 2018
Snippets
#
-- set identifiers to none for the query below to work and
-- set it back to column once it's done
set hive.support.quoted.identifiers = none;
HIVE 3
#
- BI Code typically use
db.table
- needs to change to db
.table
- Default path : /warehouse/tablespace/external/hive/default.db/test_table
Resources & Useful Links
#
ACID + HIVE