Version solving stuck in a loop when using poetry update, potentially accompanied by memory leak

[mishok13]

Description

There is a peculiar runaway loop which I'm trying to pin the cause of. Here are the steps required to reproduce it:

Prepare repo

$ cat <<EOF > pyproject.toml
[tool.poetry]
name = "foobar"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
packages = [{include = "foobar", from = "src"}]

[tool.poetry.dependencies]
python = "^3.11"
# The packages causing trouble
opentelemetry-api = "1.24.0"
opentelemetry-sdk = "*"
opentelemetry-distro = "*"
opentelemetry-exporter-otlp-proto-grpc = "*"
opentelemetry-instrumentation-urllib3 = "*"
opentelemetry-instrumentation-httpx = "*"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
EOF

Lock dependencies

❯ poetry lock
Updating dependencies
Resolving dependencies... (1.8s)

Writing lock file

Attempt updating

poetry add --no-interaction -vvv --lock 'opentelemetry-api==1.25.0'
# [skipped for brevity]
  14: ! which is caused by "opentelemetry-sdk (>=1.12,<1.13.0) requires opentelemetry-semantic-conventions (0.33b0)"
  14: ! thus: opentelemetry-distro (!=0.45b0) requires opentelemetry-instrumentation (0.21b0 || 0.22b0 || 0.23b0 || 0.23b2 || 0.24b0 || 0.25b0 || 0.25b1 || 0.25b2 || 0.26b1 || 0.27b0 || 0.28b0 || 0.28b1 || 0.29b0 || 0.30b0 || 0.30b1) or opentelemetry-api (0.17b0 || 1.0.0rc1 || 1.0.0 || 1.1.0) or opentelemetry-semantic-conventions (0.33b0) or opentelemetry-sdk (1.12.0rc1 || 1.12.0rc2 || >=1.13,<2.0)

These two last lines are then repeated infinitely until poetry process is killed. The rate is roughly 2 thousand lines of that per second on my 2 year old Macbook Pro. We have observed identical behavior in our Kubernetes environment using Debian-based images, where similar command is issued by Renovate.

Importantly Poetry seems to be leaking memory while being stuck in this loop, as we have observed Poetry OOM'ed on a beefy EC2 instance with 32GB of RAM. I haven't been able to reproduce this behavior on MacOS, as the memory use would jump between 300MB and 3.5GB. My memory profiling skills are rusty and my knowledge of Poetry internals is poor, so I don't know if I can provide any further information.

I've been able to reproduce this issue with:

• Both Homebrew and pipx-installed Poetry
• On Poetry versions 1.7.0 and 1.8.3
• On Python 3.10, 3.11 and 3.12 (installed with mise) on MacOS as well Python container images (python:3.x)

Workarounds

Workaround for this specific issue is pinning all OpenTelemetry packages using Renovate, hence not really related to Poetry.

Poetry Installation Method

pipx, Homebrew

Operating System

MacOS, Debian

Poetry Version

1.8.3, 1.7.0

Poetry Configuration

❯ poetry config --list
cache-dir = "/Users/.../Library/Caches/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = false
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/.../Library/Caches/pypoetry/virtualenvs
virtualenvs.prefer-active-python = true
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

No response

Poetry Runtime Logs

Log trimmed to the initial 3200 lines (doesn't fit in the body of the issue otherwise due to character limits) https://gist.github.com/mishok13/d0039d7152829567da3a6ea1f5b2c7d1

[dimbleby]

probably duplicate #8475 etc

[mishok13]

@dimbleby after going in detail through #8475 I suspect you are right but have no way of confirming this suspicion. I'm ok with closing this issue.

[dimbleby]

You have that power, I don't. Press the button!

[mishok13]

Ah damn, I assumed you were one of the repo maintainers. Closing now. :)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.