# t-SNE Dimensionality Reduction with Scikit-Learn

How to encode an image dataset to reduce its dimensionality and visualize it in the 2D space.

# NumPy 3d translation matrix

Here's how to translate 3d points in Python using a translation matrix.

To translate a series of points in three dimensions in Cartesian space (x, y, z) you first need to "homogenize" the points by adding a value to their projective dimension—which we'll set to one to maintain the point's original coordinates, and then multiply our point cloud using NumPy's `np.matmul` method by a transformation matrix constructed from a (4, 4) identity matrix with three translation parameters in its bottom row (tx, ty, tz).

## Steps

Here's a breakdown of the steps.

• Import the NumPy Python library
• Define a point cloud with Cartesian coordinates (x, y, z)
• Convert the points to homogeneous coordinates (x, y, z, w)
• Define our translation parameters (tx, ty, tz)
• Construct the translation matrix
• Multiply the homogenized point cloud by the transformation matrix with NumPy's `np.matmul`

## Code

``````# translate.py
import numpy as np

# Define a set of Cartesian (x, y, z) points
point_cloud = [
[0, 0, 0],
[1, 0, 0],
[0, 1, 0],
[0, 0, 1],
[1, 1, 1],
[1, 2, 3],
]

# Convert to homogeneous coordinates
point_cloud_homogeneous = []
for point in point_cloud:
point_homogeneous = point.copy()
point_homogeneous.append(1)
point_cloud_homogeneous.append(point_homogeneous)

# Define the translation
tx = 2
ty = 10
tz = 100

# Construct the translation matrix
translation_matrix = [
[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 1, 0],
[tx, ty, tz, 1],
]

# Apply the transformation to our point cloud
translated_points = np.matmul(
point_cloud_homogeneous,
translation_matrix)

# Convert to cartesian coordinates
translated_points_xyz = []
for point in translated_points:
point = np.array(point[:-1])
translated_points_xyz.append(point)

# Map original to translated point coordinates
# (x0, y0, z0) → (x1, y1, z1)
for i in range(len(point_cloud)):
point = point_cloud[i]
translated_point = translated_points_xyz[i]
print(f'{point} → {list(translated_point)}')
``````

# [Solved] Object of type ndarray is not JSON serializable

If you try to serialize a NumPy array to JSON in Python, you'll get the error below.

``````TypeError: Object of type ndarray is not JSON serializable
``````

Luckily, NumPy has a built-in method to convert one- or multi-dimensional arrays to lists, which are in turn JSON serializable.

``````import numpy as np
import json

arr = np.array([[100,200],[300,400]])

# Convert the array to list
arr_as_list = arr.tolist()

# Serialize as JSON
json.dumps(arr_as_list)
# '[[100, 200], [300, 400]]'
``````

# Return a NumPy ndarray from AWS Lambda

Here's the error I was getting when trying to return a NumPy `ndarray` in the response body of an AWS Lambda function.

``````Object of type ndarray is not JSON serializable
``````

## Reproduce the error

``````import numpy as np
import json

# A NumPy array
arr = np.array([[1,2,3],[4,5,6]])
.astype(np.float64)

# Serialize the array
json.dumps(arr)
# TypeError: Object of type ndarray is not JSON serializable
``````

## Solution

NumPy arrays provide a built-in method to convert them to lists called `.tolist()`.

``````import numpy as np
import json

# A NumPy array
arr = np.array([[1,2,3],[4,5,6.78]])
.astype(np.float64)

# Convert the NumPy array to a list
arr_as_list = arr.tolist()

# Serialize the list
json.dumps(arr_as_list)
``````

# TensorFlow with Docker in 5 minutes

How to use TensorFlow inside of a Docker container.

# Anaconda slows down Terminal's startup time on Apple Silicon

I've had `conda`'s initialization code in my `.zshrc` file for a long time. I've used it on my former Intel and M1 Macs, but it was just recently that I migrated my code to a new M1 Max Apple Silicon Mac. When I start a new Terminal window, I see how a Python process takes up to 5–10 seconds to finalize before the Terminal becomes responsive. I'm used to hitting `CMD + N` for a new window or `CMT + T` for a new tab and starting to type immediately. But this issue breaks my workflow and keeps me hanging for a few seconds per new window (!).

Here's my initialization code, auto-generated by Anaconda.

``````# >>> conda initialize >>>
!! Contents within this block are managed by 'conda init' !!
__conda_setup="\$('/Users/nono/anaconda3/bin/conda' 'shell.zsh' 'hook' 2> /dev/null)"
if [ \$? -eq 0 ]; then
eval "\$__conda_setup"
else
if [ -f "/Users/nono/anaconda3/etc/profile.d/conda.sh" ]; then
. "/Users/nono/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/Users/nono/anaconda3/bin:\$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
``````

This issue also happens with Miniforge on Macbooks, as seen in this GitHub issue titled Slow zsh startup on MacBook Pro 14-inch (M1 Pro). In my case, it's not the M1 Pro but the M1 Max. So the issue seems independent of specific chips but may be an Apple-Silicon-only problem.

Others may see this in M1, M1 Pro, M1 Max, M1 Ultra, and M2 chips and the incoming M2 Pro, M2 Max, and M2 Ultra.

Please let me know on Twitter if you find out how to make this initialization faster. In the meantime, I've removed this code and will have to get it back when I use conda, or simply find another way to initialize Anaconda on demand, only in the Terminal instances I want to use it.

# Convert from Camel Case to Snake Case with Regular Expressions in Python

Here's how to convert a string from `CamelCase` to `snake_case` in Python with regular expressions.

``````import re

# Option 1
regex = r'(?<!^)(?=[A-Z])'
re.sub(regex, '_', 'GettingSimple', 0).lower()
# returns 'getting_simple'

# Option 2
pattern = re.compile(r'(?<!^)(?=[A-Z])')
pattern.sub('_', 'nonoMartinezAlonso').lower()
# returns 'nono_martinez_alonso'
``````

See how to Convert from snake_case to camelCase.

# Python "in" and "not in" operators

Here's an example on how to use the "in" and "not in" Python operators.

``````› python
Python 3.9.13 (main, May 24 2022, 21:28:31)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
>>> names = ['James', 'Paul', 'Lily', 'Glory']
>>> names
['James', 'Paul', 'Lily', 'Glory']
>>> print('YES' if 'Lily' in names else 'NO')
YES
>>> print('YES' if 'John' in names else 'NO')
NO
>>> print('NO' if 'Lily' not in names else 'YES')
YES
>>> print('NO' if 'John' not in names else 'YES')
NO
``````

You could use this as a conditional in your code.

``````names = ['James', 'Paul', 'Lily', 'Glory']
new_person = 'Nono'

if new_person not in names:
names.append(new_person)

if new_person in names:
print(f'{new_person} was correctly added to names.')
# Nono was correctly added to names.
``````

# ucwords in Python

PHP's `ucwords()` function converts a phrase to title case.

``````// PHP
ucwords('a big snake')
// Returns "A Big Snake"
``````

You can obtain the same result in Python with the `.title()` string method.

``````# Python
"a big snake".title()
# Returns "A Big Snake"
``````

Here's how to test this function in the command-line interface.

``````python -c "print('nono martinez alonso'.title())"
``````

# Read Text from a File with Python

Here's how to read text from a file in Python; maybe a file that already exists or a file to which you wrote the text with Python.

``````file = open('/your/file.txt', 'r')

# Print the contents
print(contents)
``````

## Read file contents as lines

``````file = open('/your/file.txt', 'r')

# Read the lines of the file

# Iterate through the lines
for line in lines:
print(line)
``````

# I can't create Python 2 environments with conda on an Apple Silicon Mac

Today I tried to do this on my 13-inch MacBook Pro (M1, 2020).

``````conda create -n py2 python=2.7 -y
``````

And I continue getting this error.

``````Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json,
will retry with next repodata source.
Solving environment: failed

PackagesNotFoundError: The following packages are not available
from current channels:

- python=2.7

Current channels:

- https://conda.anaconda.org/conda-forge/osx-arm64
- https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda
package you're looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.
``````

I can create environments with Python 3 versions without a problem though; say, Python 3.7, 3.8, or 3.9.

``````conda create -n py2 python=3.9 -y
``````

# Get Python binary executable location in Python

Here's how to determine the location of the active Python binary.

``````import sys

# Get Python binary's location.
print(sys.executable)
# /Users/nono/miniforge3/bin/python
``````

# Get Python version in Python

Here's how to get the version of your Python executable with Python code and determine the location of the active Python binary.

``````import sys

# Get Python's version.
print(sys.version)
# 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:24:02) \n[Clang 11.1.0 ]

# Get Python's version information.
print(sys.version_info)
# sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0)

# Get Python binary's location.
print(sys.executable)
# /Users/nono/miniforge3/bin/python
``````

# [Solved] ValueError: invalid literal for int() with base 10

If we try to convert a literal string with decimal points—say, `'123.456'`—to an integer, we'll get this error.

``````>>> int('123.456') # Returns 123
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '123.456'
``````

The solution is to convert the string literal into a `float` first and then convert it into an `integer`.

``````int(float('123.456')) # Returns 123
``````

# How to check Python types

Here's an easy way to check your Python variable types.

We first define our variables.

``````name = "Nono"
names = ["Nono", "Bea"]
person = {"name": "Nono", "location": "Spain"}
pair = ("Getting", "Simple")
``````

Use the `isinstance()` method to check their types.

``````# Is name a string?
isinstance(name, str) # True

# Is names a list?
isinstance(names, list) # True

# Is person a dictionary?
isinstance(person, dict) # True

# Is pair a tuple?
isinstance(pair, tuple) # True

# Is name a list?
isinstance(name, list) # False

# Is person a string?
isinstance(person, str) # False
``````

Then we can build conditional statements based on a variable's type.

``````if isinstance(name, str):
print(f'Hello, {name}!')
``````

# Copy files with shutil

Here's a straightforward way to copy a file with Python to a different path or directory, also useful to duplicate or version a file with a different name.

``````import shutil

from = '/path/to/origin/file.md'
to = '/path/to/destination/new-name.md'

shutil.copy(from, to)
``````

Source

# [Solved] Python rmdir throws OSError: [Errno 66] Directory not empty:

If you're trying to remove a directory using the `os.rmdir` function, but it contains other files, you'll probably hit the following error.

``````OSError: [Errno 66] Directory not empty:
``````

You can ignore this error by using the `shutil` library instead of `os`.

``````import shutil
shutil.rmtree(path)
``````

Note that Python won't prompt you to confirm this deletion action and this may lead to deleting files by mistake.

# Map a Color in a Pillow Image to Another Color NumPy

Here's a way to map a given color in a Pillow image (PIL.Image) to another color. This is not the fastest method and it will only replace exact matches.

In this example, we're turning all black pixels in the input image (0,0,0) with blue (0,0,255).

``````from PIL import Image
import numpy as np

img = Image.open('/path/to/image.png')
img[np.where((img==[0,0,0]).all(axis=2))] = [0,0,255]
img.show()
``````

# Crop an Image with Pillow

``````from PIL import Image

img = Image.open('/path/to/image')

left = 10
top = 20
right = 10
bottom = 20

img = img.crop((left, top, right, bottom))
``````

# Load a JSON File with Python

``````import json

file = open('my-file.json')
file.close()
``````

# How to import PyTorch in Google Colab

``````import torch
print(torch.__version__)
``````

# Read Environment Variables in a Python Notebook

To read environment variables from a Python script or a Jupyter notebook, you would use this code—assuming you have a `.env` file in the directory where your script or notebook lives.

``````# .env
FOO=BAR
S3_BUCKET=YOURS3BUCKET
S3_SECRET_KEY=YOURSECRETKEYGOESHERE
``````
``````# script.py
import os
print(os.environ.get('FOO')) # Empty
``````

But this won't return the value of the environment variables, though, as you need to parse the contents of your `.env` file first.

For that, you can either use `python-dotenv`.

``````pip install python-dotenv
``````

``````# Example from https://pypi.org/project/python-dotenv/

# OR, the same with increased verbosity

# OR, explicitly providing path to '.env'
from pathlib import Path  # Python 3.6+ only
env_path = Path('.') / '.env'

# Print variable FOO
print(os.environ.get('FOO')) # Returns 'BAR'
``````

Or load the variables manually with this script.

``````import os
env_vars = !cat ../script/.env
for var in env_vars:
key, value = var.split('=')
os.environ[key] = value

# Print variable FOO
print(os.environ.get('FOO')) # Returns 'BAR'
``````

# [Solved] ERROR: Could not find a version that satisfies the requirement tensorflow

I got this error while trying to `pip3 install tensorflow`. I tried `python3 -m pip install tensorflow` as well — it didn't work.

``````ERROR: Could not find a version that satisfies the requirement tensorflow
ERROR: No matching distribution found for tensorflow
``````

As was my case, the reason for this error might be that you are using `pip` from a Python version not yet supported by any version of TensorFlow. I was running Python 3.9 and TensorFlow only had compatibility up to Python 3.8. By creating a new environment with Python 3.8 (or reverting the current environment to use 3.8) I could `pip3 install tensorflow` successfully.

# Pylance: Fast, feature-rich language support for Python in Visual Studio Code

About six months ago, Microsoft launched Pylance, a "fast and feature-rich language support for Python," available in the Visual Studio Code marketplace.

Pylance depends on our core Python extension and builds upon that experience, for those of you who have already installed it.

Among its main features are type information, auto-imports, multi-root workspace support, and type checking diagnostics.

The name Pylance serves as a nod to Monty Python’s Lancelot, who is the first knight to answer the bridgekeeper’s questions in the Holy Grail.

# Write Text to File with Python

To write (or save) text to a file using Python, you can either append text or overwrite all existing contents with new text.

## Appending text

To append text, open the file in `append` mode, write to it to add lines of text, and close it.

``````file = open('/path/to/file.txt', 'a') # 'a' is append-to-end-of-file mode
file.close()
``````

## Overwriting text

You can also write the entire contents of the files, overwriting any existing content using the `w` mode instead of `a`.

``````file = open('/path/to/file.txt', 'w') # 'w' is overwrite mode
file.write('This will override any existing content in the text to this document.')
file.close()
``````

## Line breaks

You can use `\r\n` or `\n` and other codes to add line breaks to your document.

``````file = open('/path/to/file.txt', 'w') # 'w' is overwrite mode
file.write('First line.\nSecond line.\nThird line.\n\nNono.MA')
file.close()
``````
``````# file.txt
First line.
Second line.
Third line.

Nono.MA
``````

# Python File Exists

To determine whether a file or directory exists using Python you can use either the `os.path` or the `pathlib` library.

The `os` library offers three methods: `path.exists`, `path.isfile`, and `path.isdir`.

``````import os

# Returns True if file or dir exists
os.path.exists('/path/to/file/or/dir')

# Returns True if exists and is a file
os.path.isfile('/path/to/file/or/dir')

# Returns True if exists and is a directory
os.path.isdir()
``````

The `pathlib` library has many methods (not covered here) but the `pathlib.Path('/path/to/file').exists()` also does the job.

``````import pathlib

file = pathlib.Path('/path/to/file')

# Returns True if file or dir exists
file.exists()
``````

# Resize Segmentation Masks with TensorFlow

When manipulating semantic segmentation datasets, I found myself having to downsize segmentation masks without adding extra colors. If the image is cleanly encoded as a PNG, only the colors representing each of the classes contained in the label map will be present, and no antialias intermediate colors will exist in the image.

When resizing, though, antialias might add artifacts to your images to soften the edges, adding new colors that don't belong to any class in the label map. We can overcome this problem loading (or decoding) input images with TensorFlow as PNG and resizing our images with TensorFlow's `NEAREST_NEIGHBOR` resizing method.

(You can find a list of all TensorFlow's resize methods here, and an explanation of what each of them does here.)

``````import tensorflow as tf

# Decode as PNG
img = tf.io.decode_png(
img,
channels=3,
dtype=tf.uint8
)

# Resize using nearest neighbor to avoid adding new colors
# For that purpose, antialias is ignored with this resize method
img = tf.image.resize(
img,
(128, 128), # (width, height)
antialias=False, # Ignored when using NEAREST_NEIGHBOR
method=tf.image.ResizeMethod.NEAREST_NEIGHBOR
)

# Save the resize image back to PNG
tf.keras.preprocessing.image.save_img(
'/path/to/output/image.png',
img
)
``````

# What's the difference between code linters and formatters?

## TL;DR

• Linters flag bugs and bad practices
• Formatters fix your code to match style guides

## Linters

Code linters analyze code statically to flag programming errors, catch bugs, stylistic errors, and suspicious constructs,1 using the abstract syntax tree or AST. Code linting promotes and enforces best practices by ensuring your code matches your style guide.2

You can expect a linter to warn you of functions whose complexity needs to be reduced, syntax improvements, code practices that go against configured or standard conventions, etc.

For instance, eslint is a widely-used JavaScript linter and SonarLint is an IDE extension that you can use for linting code in VSCode.

For illustration purposes, here's a sample code cognitive complexity warning from SonarLint for a Python function.

``````Refactor this function to reduce its Cognitive Complexity
from 37 to the 15 allowed. sonarlint(python:S3776)
``````

## Formatters

Code formatters fix style—spacing, line jumps, comments—which helps enforce programming and formatting rules that can be easily automated, which helps reduce future code diffs by delegating formatting concerns to an automatic tool rather than individual developers.

For instance, autopep8 automatically formats Python code to conform to the PEP 8 style guide.

As an example, take a look at the contents of this JSON file.

``````// names.json
{"names": ["James", "Claire", "Peter", "Lily"]}
``````

By right-clicking on a JSON file with these contents on Visual Studio Code, you can select `Format Document` or press `⌥` + `⇧` + `F` (on macOS) to obtain the following results.

``````// names.json
{
"names": [
"James",
"Claire",
"Peter",
"Lily"
]
}
``````

1. Lint. Wikipedia. Accessed July 8, 2022.

2. Clean code linters. GitHub. Accessed July 8, 2022.

# Make Python 3 the Default on macOS

While `macOS` ships with Python 2 by default, you can install set Python 3 as the default Python version on your Mac.

First, you install Python 3 with Homebrew.

``````brew update && brew install python
``````

To make this new version your default, you can add the following line to your `~/.zshrc` file (or `~/.bashrc` if you want to expose it in `bash` instead of `zsh`).

``````alias python=/usr/local/bin/python3
``````

Then open a new Terminal and Python 3 should be running.

Let's verify this is true.

``````python --version # e.g. Python 3.8.5
``````

## How do I find the `python3` path?

Homebrew provides info about any installed "bottle" via the `info` command.

``````brew info python
# python@3.8: stable 3.8.5 (bottled)
# Interpreted, interactive, object-oriented programming language
# https://www.python.org/
# /usr/local/Cellar/python@3.8/3.8.5 (4,372 files, 67.7MB) *
# ...
``````

And you can find the path we're looking for `grep`.

``````brew info python | grep bin
# /usr/local/bin/python3
# /usr/local/opt/python@3.8/libexec/bin
``````

## Another way

You can also symlink `python3` to `python`.

``````ln -sf /usr/local/bin/python3 /usr/local/bin/python
``````

In case your `/usr/local/bin/python3` is also symlinked, you can check where it's symlinked to with:

``````readlink /usr/local/bin/python3
``````

In my case, it returns `../Cellar/python@3.9/3.9.1_6/bin/python3`.

## How do I use Python 2 if I need it?

Your system's Python 2.7 is still there.

``````/usr/bin/python --version # e.g Python 2.7.16
``````

You can also use Homebrew's Python 2.

``````brew install python@2
``````

Want to see older publications? Visit the archive.

Listen to Getting Simple .