Anaconda slows down Terminal's startup time on Apple Silicon

I've had conda's initialization code in my .zshrc file for a long time. I've used it on my former Intel and M1 Macs, but it was just recently that I migrated my code to a new M1 Max Apple Silicon Mac. When I start a new Terminal window, I see how a Python process takes up to 5–10 seconds to finalize before the Terminal becomes responsive. I'm used to hitting CMD + N for a new window or CMT + T for a new tab and starting to type immediately. But this issue breaks my workflow and keeps me hanging for a few seconds per new window (!).

Here's my initialization code, auto-generated by Anaconda.

# >>> conda initialize >>>
!! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/nono/anaconda3/bin/conda' 'shell.zsh' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/Users/nono/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/Users/nono/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/Users/nono/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

This issue also happens with Miniforge on Macbooks, as seen in this GitHub issue titled Slow zsh startup on MacBook Pro 14-inch (M1 Pro). In my case, it's not the M1 Pro but the M1 Max. So the issue seems independent of specific chips but may be an Apple-Silicon-only problem.

Others may see this in M1, M1 Pro, M1 Max, M1 Ultra, and M2 chips and the incoming M2 Pro, M2 Max, and M2 Ultra.

Please let me know on Twitter if you find out how to make this initialization faster. In the meantime, I've removed this code and will have to get it back when I use conda, or simply find another way to initialize Anaconda on demand, only in the Terminal instances I want to use it.