Nono.MA

NOVEMBER 21, 2022


How to encode an image dataset to reduce its dimensionality and visualize it in the 2D space.


See transcript ›

NOVEMBER 16, 2022

Here's how to translate 3d points in Python using a translation matrix.


To translate a series of points in three dimensions in Cartesian space (x, y, z) you first need to "homogenize" the points by adding a value to their projective dimension—which we'll set to one to maintain the point's original coordinates, and then multiply our point cloud using NumPy's np.matmul method by a transformation matrix constructed from a (4, 4) identity matrix with three translation parameters in its bottom row (tx, ty, tz).

Steps

Here's a breakdown of the steps.

  • Import the NumPy Python library
  • Define a point cloud with Cartesian coordinates (x, y, z)
  • Convert the points to homogeneous coordinates (x, y, z, w)
  • Define our translation parameters (tx, ty, tz)
  • Construct the translation matrix
  • Multiply the homogenized point cloud by the transformation matrix with NumPy's np.matmul

Code

# translate.py
import numpy as np

# Define a set of Cartesian (x, y, z) points
point_cloud = [
    [0, 0, 0],
    [1, 0, 0],
    [0, 1, 0],
    [0, 0, 1],
    [1, 1, 1],
    [1, 2, 3],
]

# Convert to homogeneous coordinates
point_cloud_homogeneous = []
for point in point_cloud:
    point_homogeneous = point.copy()
    point_homogeneous.append(1)
    point_cloud_homogeneous.append(point_homogeneous)

# Define the translation
tx = 2
ty = 10
tz = 100

# Construct the translation matrix
translation_matrix = [
    [1, 0, 0, 0],
    [0, 1, 0, 0],
    [0, 0, 1, 0],
    [tx, ty, tz, 1],
]

# Apply the transformation to our point cloud
translated_points = np.matmul(
    point_cloud_homogeneous,
    translation_matrix)

# Convert to cartesian coordinates
translated_points_xyz = []
for point in translated_points:
    point = np.array(point[:-1])
    translated_points_xyz.append(point)

# Map original to translated point coordinates
# (x0, y0, z0) → (x1, y1, z1)
for i in range(len(point_cloud)):
    point = point_cloud[i]
    translated_point = translated_points_xyz[i]
    print(f'{point} → {list(translated_point)}')

NOVEMBER 12, 2022

If you try to serialize a NumPy array to JSON in Python, you'll get the error below.

TypeError: Object of type ndarray is not JSON serializable

Luckily, NumPy has a built-in method to convert one- or multi-dimensional arrays to lists, which are in turn JSON serializable.

import numpy as np
import json

# Define your NumPy array
arr = np.array([[100,200],[300,400]])

# Convert the array to list
arr_as_list = arr.tolist()

# Serialize as JSON
json.dumps(arr_as_list)
# '[[100, 200], [300, 400]]'

NOVEMBER 11, 2022

Here's the error I was getting when trying to return a NumPy ndarray in the response body of an AWS Lambda function.

Object of type ndarray is not JSON serializable

Reproduce the error

import numpy as np
import json

# A NumPy array
arr = np.array([[1,2,3],[4,5,6]])
        .astype(np.float64)

# Serialize the array
json.dumps(arr)
# TypeError: Object of type ndarray is not JSON serializable

Solution

NumPy arrays provide a built-in method to convert them to lists called .tolist().

import numpy as np
import json

# A NumPy array
arr = np.array([[1,2,3],[4,5,6.78]])
        .astype(np.float64)

# Convert the NumPy array to a list
arr_as_list = arr.tolist()

# Serialize the list
json.dumps(arr_as_list)

NOVEMBER 9, 2022


How to use TensorFlow inside of a Docker container.


See transcript ›

OCTOBER 1, 2022

I've had conda's initialization code in my .zshrc file for a long time. I've used it on my former Intel and M1 Macs, but it was just recently that I migrated my code to a new M1 Max Apple Silicon Mac. When I start a new Terminal window, I see how a Python process takes up to 5–10 seconds to finalize before the Terminal becomes responsive. I'm used to hitting CMD + N for a new window or CMT + T for a new tab and starting to type immediately. But this issue breaks my workflow and keeps me hanging for a few seconds per new window (!).

Here's my initialization code, auto-generated by Anaconda.

# >>> conda initialize >>>
!! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/nono/anaconda3/bin/conda' 'shell.zsh' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/Users/nono/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/Users/nono/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/Users/nono/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

This issue also happens with Miniforge on Macbooks, as seen in this GitHub issue titled Slow zsh startup on MacBook Pro 14-inch (M1 Pro). In my case, it's not the M1 Pro but the M1 Max. So the issue seems independent of specific chips but may be an Apple-Silicon-only problem.

Others may see this in M1, M1 Pro, M1 Max, M1 Ultra, and M2 chips and the incoming M2 Pro, M2 Max, and M2 Ultra.

Please let me know on Twitter if you find out how to make this initialization faster. In the meantime, I've removed this code and will have to get it back when I use conda, or simply find another way to initialize Anaconda on demand, only in the Terminal instances I want to use it.

SEPTEMBER 3, 2022

Here's how to convert a string from CamelCase to snake_case in Python with regular expressions.

import re

# Option 1
regex = r'(?<!^)(?=[A-Z])'
re.sub(regex, '_', 'GettingSimple', 0).lower()
# returns 'getting_simple'

# Option 2
pattern = re.compile(r'(?<!^)(?=[A-Z])')
pattern.sub('_', 'nonoMartinezAlonso').lower()
# returns 'nono_martinez_alonso'

See how to Convert from snake_case to camelCase.

JULY 17, 2022

Here's an example on how to use the "in" and "not in" Python operators.

› python
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> names = ['James', 'Paul', 'Lily', 'Glory']
>>> names
['James', 'Paul', 'Lily', 'Glory']
>>> print('YES' if 'Lily' in names else 'NO')
YES
>>> print('YES' if 'John' in names else 'NO')
NO
>>> print('NO' if 'Lily' not in names else 'YES')
YES
>>> print('NO' if 'John' not in names else 'YES')
NO

You could use this as a conditional in your code.

names = ['James', 'Paul', 'Lily', 'Glory']
new_person = 'Nono'

if new_person not in names:
  names.append(new_person)
  print(f'Added {new_person} to names.')
  # Added Nono to names.

if new_person in names:
  print(f'{new_person} was correctly added to names.')
  # Nono was correctly added to names.

JULY 8, 2022

PHP's ucwords() function converts a phrase to title case.

// PHP
ucwords('a big snake') 
// Returns "A Big Snake"

You can obtain the same result in Python with the .title() string method.

# Python
"a big snake".title()
# Returns "A Big Snake"

Here's how to test this function in the command-line interface.

python -c "print('nono martinez alonso'.title())"

JUNE 30, 2022

Here's how to read text from a file in Python; maybe a file that already exists or a file to which you wrote the text with Python.

Read file contents

file = open('/your/file.txt', 'r')

# Read all file contents
contents = file.read()

# Print the contents
print(contents)

Read file contents as lines

file = open('/your/file.txt', 'r')

# Read the lines of the file
lines = file.readlines()

# Iterate through the lines
for line in lines:
  print(line)

JUNE 16, 2022

Today I tried to do this on my 13-inch MacBook Pro (M1, 2020).

conda create -n py2 python=2.7 -y

And I continue getting this error.

Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json,
will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available
from current channels:

  - python=2.7

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda
package you're looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

I can create environments with Python 3 versions without a problem though; say, Python 3.7, 3.8, or 3.9.

conda create -n py2 python=3.9 -y

JUNE 3, 2022

Here's how to determine the location of the active Python binary.

import sys

# Get Python binary's location.
print(sys.executable)
# /Users/nono/miniforge3/bin/python

See how to get Python's version information.

JUNE 2, 2022

Here's how to get the version of your Python executable with Python code and determine the location of the active Python binary.

import sys

# Get Python's version.
print(sys.version)
# 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:24:02) \n[Clang 11.1.0 ]

# Get Python's version information.
print(sys.version_info)
# sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0)

# Get Python binary's location.
print(sys.executable)
# /Users/nono/miniforge3/bin/python

APRIL 27, 2022

If we try to convert a literal string with decimal points—say, '123.456'—to an integer, we'll get this error.

>>> int('123.456') # Returns 123
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '123.456'

The solution is to convert the string literal into a float first and then convert it into an integer.

int(float('123.456')) # Returns 123

APRIL 14, 2022

Here's an easy way to check your Python variable types.

We first define our variables.

name = "Nono"
names = ["Nono", "Bea"]
person = {"name": "Nono", "location": "Spain"}
pair = ("Getting", "Simple")

Use the isinstance() method to check their types.

# Is name a string?
isinstance(name, str) # True

# Is names a list?
isinstance(names, list) # True

# Is person a dictionary?
isinstance(person, dict) # True

# Is pair a tuple?
isinstance(pair, tuple) # True

# Is name a list?
isinstance(name, list) # False

# Is person a string?
isinstance(person, str) # False

Then we can build conditional statements based on a variable's type.

if isinstance(name, str):
  print(f'Hello, {name}!')

MARCH 7, 2022

Here's a straightforward way to copy a file with Python to a different path or directory, also useful to duplicate or version a file with a different name.

import shutil

from = '/path/to/origin/file.md'
to = '/path/to/destination/new-name.md'

shutil.copy(from, to)

Source

SEPTEMBER 21, 2021

If you're trying to remove a directory using the os.rmdir function, but it contains other files, you'll probably hit the following error.

OSError: [Errno 66] Directory not empty:

You can ignore this error by using the shutil library instead of os.

import shutil
shutil.rmtree(path)

Note that Python won't prompt you to confirm this deletion action and this may lead to deleting files by mistake.

MAY 13, 2021

Here's a way to map a given color in a Pillow image (PIL.Image) to another color. This is not the fastest method and it will only replace exact matches.

In this example, we're turning all black pixels in the input image (0,0,0) with blue (0,0,255).

from PIL import Image
import numpy as np

img = Image.open('/path/to/image.png')
img[np.where((img==[0,0,0]).all(axis=2))] = [0,0,255]
img.show()

MAY 13, 2021

from PIL import Image

img = Image.open('/path/to/image')

left = 10
top = 20
right = 10
bottom = 20

img = img.crop((left, top, right, bottom))

LAST UPDATED MARCH 28, 2022

import json

file = open('my-file.json')
obj = json.load(file)
file.close()

APRIL 28, 2021

import torch
print(torch.__version__)

FEBRUARY 24, 2021

To read environment variables from a Python script or a Jupyter notebook, you would use this code—assuming you have a .env file in the directory where your script or notebook lives.

# .env
FOO=BAR
S3_BUCKET=YOURS3BUCKET
S3_SECRET_KEY=YOURSECRETKEYGOESHERE
# script.py
import os
print(os.environ.get('FOO')) # Empty

But this won't return the value of the environment variables, though, as you need to parse the contents of your .env file first.

For that, you can either use python-dotenv.

pip install python-dotenv

Then use this Python library to load your variables.

# Example from https://pypi.org/project/python-dotenv/
from dotenv import load_dotenv
load_dotenv()

# OR, the same with increased verbosity
load_dotenv(verbose=True)

# OR, explicitly providing path to '.env'
from pathlib import Path  # Python 3.6+ only
env_path = Path('.') / '.env'
load_dotenv(dotenv_path=env_path)

# Print variable FOO
print(os.environ.get('FOO')) # Returns 'BAR'

Or load the variables manually with this script.

import os
env_vars = !cat ../script/.env
for var in env_vars:
    key, value = var.split('=')
    os.environ[key] = value

# Print variable FOO
print(os.environ.get('FOO')) # Returns 'BAR'

JANUARY 13, 2021

I got this error while trying to pip3 install tensorflow. I tried python3 -m pip install tensorflow as well — it didn't work.

ERROR: Could not find a version that satisfies the requirement tensorflow
ERROR: No matching distribution found for tensorflow

As was my case, the reason for this error might be that you are using pip from a Python version not yet supported by any version of TensorFlow. I was running Python 3.9 and TensorFlow only had compatibility up to Python 3.8. By creating a new environment with Python 3.8 (or reverting the current environment to use 3.8) I could pip3 install tensorflow successfully.

JANUARY 8, 2021

About six months ago, Microsoft launched Pylance, a "fast and feature-rich language support for Python," available in the Visual Studio Code marketplace.

Pylance depends on our core Python extension and builds upon that experience, for those of you who have already installed it.

Among its main features are type information, auto-imports, multi-root workspace support, and type checking diagnostics.

The name Pylance serves as a nod to Monty Python’s Lancelot, who is the first knight to answer the bridgekeeper’s questions in the Holy Grail.

OCTOBER 13, 2020

To write (or save) text to a file using Python, you can either append text or overwrite all existing contents with new text.

Appending text

To append text, open the file in append mode, write to it to add lines of text, and close it.

file = open('/path/to/file.txt', 'a') # 'a' is append-to-end-of-file mode
file.write('Adding text to this document.')
file.close()

Overwriting text

You can also write the entire contents of the files, overwriting any existing content using the w mode instead of a.

file = open('/path/to/file.txt', 'w') # 'w' is overwrite mode
file.write('This will override any existing content in the text to this document.')
file.close()

Line breaks

You can use \r\n or \n and other codes to add line breaks to your document.

file = open('/path/to/file.txt', 'w') # 'w' is overwrite mode
file.write('First line.\nSecond line.\nThird line.\n\nNono.MA')
file.close()
# file.txt
First line.
Second line.
Third line.

Nono.MA

OCTOBER 8, 2020

To determine whether a file or directory exists using Python you can use either the os.path or the pathlib library.

The os library offers three methods: path.exists, path.isfile, and path.isdir.

import os

# Returns True if file or dir exists
os.path.exists('/path/to/file/or/dir')

# Returns True if exists and is a file
os.path.isfile('/path/to/file/or/dir')

# Returns True if exists and is a directory
os.path.isdir()

The pathlib library has many methods (not covered here) but the pathlib.Path('/path/to/file').exists() also does the job.

import pathlib

file = pathlib.Path('/path/to/file')

# Returns True if file or dir exists
file.exists()

SEPTEMBER 30, 2020

When manipulating semantic segmentation datasets, I found myself having to downsize segmentation masks without adding extra colors. If the image is cleanly encoded as a PNG, only the colors representing each of the classes contained in the label map will be present, and no antialias intermediate colors will exist in the image.

When resizing, though, antialias might add artifacts to your images to soften the edges, adding new colors that don't belong to any class in the label map. We can overcome this problem loading (or decoding) input images with TensorFlow as PNG and resizing our images with TensorFlow's NEAREST_NEIGHBOR resizing method.

(You can find a list of all TensorFlow's resize methods here, and an explanation of what each of them does here.)

import tensorflow as tf

# Read image file
img = tf.io.read_file('/path/to/input/image.png')

# Decode as PNG
img = tf.io.decode_png(
  img,
  channels=3,
  dtype=tf.uint8
)

# Resize using nearest neighbor to avoid adding new colors
# For that purpose, antialias is ignored with this resize method
img = tf.image.resize(
  img,
  (128, 128), # (width, height)
  antialias=False, # Ignored when using NEAREST_NEIGHBOR
  method=tf.image.ResizeMethod.NEAREST_NEIGHBOR
)

# Save the resize image back to PNG
tf.keras.preprocessing.image.save_img(
  '/path/to/output/image.png',
  img
)

LAST UPDATED JULY 8, 2022

TL;DR

  • Linters flag bugs and bad practices
  • Formatters fix your code to match style guides

Linters

Code linters analyze code statically to flag programming errors, catch bugs, stylistic errors, and suspicious constructs,1 using the abstract syntax tree or AST. Code linting promotes and enforces best practices by ensuring your code matches your style guide.2

You can expect a linter to warn you of functions whose complexity needs to be reduced, syntax improvements, code practices that go against configured or standard conventions, etc.

For instance, eslint is a widely-used JavaScript linter and SonarLint is an IDE extension that you can use for linting code in VSCode.

For illustration purposes, here's a sample code cognitive complexity warning from SonarLint for a Python function.

Refactor this function to reduce its Cognitive Complexity
from 37 to the 15 allowed. sonarlint(python:S3776)

Formatters

Code formatters fix style—spacing, line jumps, comments—which helps enforce programming and formatting rules that can be easily automated, which helps reduce future code diffs by delegating formatting concerns to an automatic tool rather than individual developers.

For instance, autopep8 automatically formats Python code to conform to the PEP 8 style guide.

As an example, take a look at the contents of this JSON file.

// names.json
{"names": ["James", "Claire", "Peter", "Lily"]}

By right-clicking on a JSON file with these contents on Visual Studio Code, you can select Format Document or press + + F (on macOS) to obtain the following results.

// names.json
{
    "names": [
        "James",
        "Claire",
        "Peter",
        "Lily"
    ]
}

  1. Lint. Wikipedia. Accessed July 8, 2022. 

  2. Clean code linters. GitHub. Accessed July 8, 2022. 

AUGUST 13, 2020

While macOS ships with Python 2 by default, you can install set Python 3 as the default Python version on your Mac.

First, you install Python 3 with Homebrew.

brew update && brew install python

To make this new version your default, you can add the following line to your ~/.zshrc file (or ~/.bashrc if you want to expose it in bash instead of zsh).

alias python=/usr/local/bin/python3

Then open a new Terminal and Python 3 should be running.

Let's verify this is true.

python --version # e.g. Python 3.8.5

How do I find the python3 path?

Homebrew provides info about any installed "bottle" via the info command.

brew info python
# python@3.8: stable 3.8.5 (bottled)
# Interpreted, interactive, object-oriented programming language
# https://www.python.org/
# /usr/local/Cellar/python@3.8/3.8.5 (4,372 files, 67.7MB) *
# ...

And you can find the path we're looking for grep.

brew info python | grep bin
# /usr/local/bin/python3
# /usr/local/opt/python@3.8/libexec/bin

Another way

You can also symlink python3 to python.

ln -sf /usr/local/bin/python3 /usr/local/bin/python

In case your /usr/local/bin/python3 is also symlinked, you can check where it's symlinked to with:

readlink /usr/local/bin/python3

In my case, it returns ../Cellar/python@3.9/3.9.1_6/bin/python3.

How do I use Python 2 if I need it?

Your system's Python 2.7 is still there.

/usr/bin/python --version # e.g Python 2.7.16

You can also use Homebrew's Python 2.

brew install python@2

Continue reading ›

Want to see older publications? Visit the archive.

Listen to Getting Simple .