Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consensus fails when enabling p2p tls #222

Open
NathanSavageKaimai opened this issue Aug 13, 2024 · 1 comment · May be fixed by #223
Open

Consensus fails when enabling p2p tls #222

NathanSavageKaimai opened this issue Aug 13, 2024 · 1 comment · May be fixed by #223

Comments

@NathanSavageKaimai
Copy link
Contributor

Problem

Currently, if config.cluster.p2p.enable_tls is set to true in values.yaml, the different pods in the cluster will fail to communicate with each other.

In my logs I have the following:

Starting initializing for pod 0
           _                 _    
  __ _  __| |_ __ __ _ _ __ | |_  
 / _` |/ _` | '__/ _` | '_ \| __| 
| (_| | (_| | | | (_| | | | | |_  
 \__, |\__,_|_|  \__,_|_| |_|\__| 
    |_|                           

Version: 1.10.0, build: 851f03bb
Access web UI at http://localhost:6333/dashboard

2024-08-13T11:16:44.533771Z  INFO storage::content_manager::consensus::persistent: Loading raft state from ./storage/raft_state.json    
2024-08-13T11:16:44.686031Z  INFO qdrant: Telemetry reporting enabled, id: 764cb1fe-6733-402a-9d5b-05199641d557    
2024-08-13T11:16:44.696863Z  INFO qdrant::actix: TLS disabled for REST API    
2024-08-13T11:16:44.699864Z  INFO qdrant::actix: Qdrant HTTP listening on 6333    
2024-08-13T11:16:44.704436Z  INFO actix_server::builder: Starting 11 workers
2024-08-13T11:16:44.704480Z  INFO actix_server::server: Actix runtime found; starting in Actix runtime
2024-08-13T11:16:44.711245Z  INFO qdrant::tonic: Qdrant gRPC listening on 6334    
2024-08-13T11:16:44.711306Z  INFO qdrant::tonic: TLS disabled for gRPC API    
2024-08-13T11:16:44.712137Z  INFO qdrant::tonic: TLS enabled for internal gRPC API (TTL not supported)    
2024-08-13T11:16:45.080494Z ERROR qdrant::common::health: GetConsensusCommit request failed: Error in closure supplied to transport channel pool: status: Unavailable, message: "Failed to connect to http://fido-qdrant-2.fido-qdrant-headless:6335/, error: transport error", details: [], metadata: MetadataMap { headers: {} }    
2024-08-13T11:16:45.083961Z ERROR qdrant::common::health: GetConsensusCommit request failed: Error in closure supplied to transport channel pool: status: Unavailable, message: "Failed to connect to http://fido-qdrant-1.fido-qdrant-headless:6335/, error: transport error", details: [], metadata: MetadataMap { headers: {} }    
2024-08-13T11:16:47.198819Z  WARN storage::content_manager::consensus_manager: Failed to send message to http://fido-qdrant-2.fido-qdrant-headless:6335/ with error: Error in closure supplied to transport channel pool: status: Unavailable, message: "Failed to connect to http://fido-qdrant-2.fido-qdrant-headless:6335/, error: transport error", details: [], metadata: MetadataMap { headers: {} }    
2024-08-13T11:16:47.201503Z  WARN storage::content_manager::consensus_manager: Failed to send message to http://fido-qdrant-1.fido-qdrant-headless:6335/ with error: Error in closure supplied to transport channel pool: status: Unavailable, message: "Failed to connect to http://fido-qdrant-1.fido-qdrant-headless:6335/, error: transport error", details: [], metadata: MetadataMap { headers: {} }

Possible cause

The pods are trying to communicate with http. It follows that with tls enabled they should be talking via https.

The bootstrap address and uri are set by startup flags in the config map template. They are set to use http regardless of if internal tls is enabled

Potential Solution

This could be fixed by modifying the template to query if internal tls is enabled.

I will test if this fixes the issue shortly and make a PR

@kinoute
Copy link

kinoute commented Aug 28, 2024

I can confirm that the proposed PR works. It would be nice to have it in the official Helm release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants