That is the git_prompt_info / git_prompt_status / parse_git_dirty functions. You can disable them per repository with one of these run inside the git repository:
git config oh-my-zsh.hide-info 1 # git_prompt_info
git config oh-my-zsh.hide-status 1 # git_prompt_status
git config oh-my-zsh.hide-dirty 1 # parse_git_dirty
You can also disable them entirely by either (1) redefining the function with an empty body in your zshrc file (after OMZ is sourced), or (2) running the commands above with a --global flag.
git_prompt_info(){} # this overwrites the git_prompt_info so it does nothing
git config --global oh-my-zsh.hide-info 1 # git_prompt_info
git config --global oh-my-zsh.hide-status 1 # git_prompt_status
git config --global oh-my-zsh.hide-dirty 1 # parse_git_dirty
In our experience git_prompt_status and parse_git_dirty are the ones that cause performance issues, so you can disable them and keep the git_prompt_info function to have branch info with none of the performance drawbacks.