Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

edgedl.me.gvt1.com is not working properly as of 7:40am GMT #60

Open
vekien opened this issue Oct 5, 2023 · 7 comments
Open

edgedl.me.gvt1.com is not working properly as of 7:40am GMT #60

vekien opened this issue Oct 5, 2023 · 7 comments

Comments

@vekien
Copy link

vekien commented Oct 5, 2023

Hello,

Not sure if this is the right place but it's the place that is Chrome For Testing.

We noticed this morning that downloads from edgedl.me.gvt1.com are no longer working on a range of EC2 servers this morning. We have hundreds of servers, all configured identical and about half of them cannot connect to edgedl.me.gvt1.com

curl -v https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.149/linux64/chrome-linux64.zip
*   Trying 34.104.35.123:443...
* TCP_NODELAY set
*   Trying 2600:1900:4110:86f:::443...
* TCP_NODELAY set
* Immediate connect fail for 2600:1900:4110:86f::: Cannot assign requested address

* TCP_NODELAY set
*   Trying 2600:1900:4110:86f:::443...
* TCP_NODELAY set
* Immediate connect fail for 2600:1900:4110:86f::: Network is unreachable
*   Trying 2600:1900:4110:86f:::443...
* TCP_NODELAY set
* Immediate connect fail for 2600:1900:4110:86f::: Network is unreachable
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0*   Trying 2600:1900:4110:86f:::443...
* TCP_NODELAY set

Has something changed with edgedl.me.gvt1.com ? or are there any mirrors?
@OrKoN OrKoN mentioned this issue Oct 5, 2023
Closed
@pattrykks
Copy link

In our company we have same problems. We can reach https://edgedl.me.gvt1.com from developers machine but we cannot reach it from CI runners. It looks like we were block by some kind of firewall rules on server side

@vekien
Copy link
Author

vekien commented Oct 5, 2023

In our company we have same problems. We can reach https://edgedl.me.gvt1.com from developers machine but we cannot reach it from CI runners. It looks like we were block by some kind of firewall rules on server side

this is similar. Infact even some of our AWS EC2 servers can connect. Server A will connect, Server B cannot connect. Both are 100% open to the public on AWS Firewalls, both have identical Network and VPC setups. It's very strange...

We have had to move all dockers that use this to another EC2 cluster which seems to ping, but this has only broken this morning... It's been fine for.. a very long time!

@OrKoN
Copy link
Collaborator

OrKoN commented Oct 5, 2023

@vekien are there any ongoing AWS incidents? we are also able to connect from our networks.

@RopoMen
Copy link

RopoMen commented Oct 5, 2023

It seems to be ipv6 issue, puppeteer/puppeteer#11060 (comment)

With curl use --ipv4 flag and with node / npm --dns-result-order=ipv4first

@vekien
Copy link
Author

vekien commented Oct 5, 2023

It seems to be ipv6 issue, puppeteer/puppeteer#11060 (comment)

With curl use --ipv4 flag and with node / npm --dns-result-order=ipv4first

We had issues even forcing ipv4 as well as disabling ipv6 completely.

Wonder if it's slightly separate issue or something along the routing with that url.

@vekien are there any ongoing AWS incidents? we are also able to connect from our networks.

No incident but from experience AWS rarely tell us.

In typical fashion after moving all the dockers to working servers it seems to be now working in all servers where it was previously failing, so it seemed to be short lived issue...

I noticed curl tests went from: Network is unreachable --> Cannot assign requested address --> Connected to edgedl.me.gvt1.com (34.104.35.123) port 443 (#0)

So it now works fine! I will keep monitoring over the day, maybe it was some routing issue on AWS end.

curl -v -O https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.149/linux64/chrome-linux64.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 34.104.35.123:443...
* Connected to edgedl.me.gvt1.com (34.104.35.123) port 443 (#0)
* ALPN: offers h2,http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
.....

@hjaske
Copy link

hjaske commented Oct 6, 2023 •
edited
Loading

Since Chrome dev downloads moved to edgedl.me.gvt1.com, there have been constant connection problems. In most cases the problem seems to be that TLS negotiation fails after "Client Hello". From the downloader perspective, it seems that edgedl service has backends that behave differently than others and when connection initiation starts on a problematic service node it fails.
Different connection issues seem to be commented on many issues regarding Chrome dev tool downloads after service changes, but the issue hasn't reached edgedl.me.gvt1.com service admins. Retries are required for the download to be successful.

~ Fri Oct  6 09:33:50 EEST 2023
curl --trace-time --trace-ascii - -4 -Io /dev/null https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.92/linux64/chromedriver-linux64.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     009:33:52.741048 == Info:   Trying 34.104.35.123:443...
09:33:52.766018 == Info: Connected to edgedl.me.gvt1.com (34.104.35.123) port 443 (#0)
09:33:52.766120 == Info: ALPN: offers h2,http/1.1
09:33:52.766313 == Info: (304) (OUT), TLS handshake, Client hello (1):
09:33:52.766318 => Send SSL data, 323 bytes (0x143)
0000: ...?..5.._.H.s...>.w_..2...DE.-....... x..`......&..G.:...4.Jf..
0040: {V=e..].b.............0.,.(.$.......k.9...........=.5...../.+.'.
0080: #.......g.3...E...<./...A.......................+............3.&
00c0: .$... ..^..7..........CI.....|.....E.W.........edgedl.me.gvt1.co
0100: m.......................................................h2.http/
0140: 1.1
09:33:52.773021 == Info:  CAfile: /etc/ssl/cert.pem
09:33:52.773039 == Info:  CApath: none
  0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     009:34:09.369026 == Info: LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
  0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
09:34:09.369364 == Info: Closing connection 0
curl: (35) LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure

curl 8.1.2 (x86_64-apple-darwin23.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.55.1

~ Fri Oct  6 09:32:26 EEST 2023
curl --trace-ascii - -4 -Io /dev/null https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.92/linux64/chromedriver-linux64.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0== Info:   Trying 34.104.35.123...
== Info: TCP_NODELAY set
== Info: Connected to edgedl.me.gvt1.com (34.104.35.123) port 443 (#0)
== Info: ALPN, offering h2
== Info: ALPN, offering http/1.1
== Info: successfully set certificate verify locations:
== Info:   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
=> Send SSL data, 5 bytes (0x5)
0000: .....
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
=> Send SSL data, 512 bytes (0x200)
0000: ......{..H..m.....{"Z%P...ZM...q.9E[. .D.A..9.s..:.s.&.R...... 
0040: 3F?.i...H.........,.0.......+./...#.'.................=.<.5./...
0080: ........k.g.9.3.....k.........edgedl.me.gvt1.com................
00c0: ........3t.........h2.http/1.1.........1.....&.$................
0100: .....................+........-.....3.&.$... %.yF...2...,5 .D.R.
0140: .[..;@?...[[s...................................................
0180: ................................................................
01c0: ................................................................
  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0<= Recv SSL data, 5 bytes (0x5)
0000: .....
== Info: TLSv1.3 (IN), TLS alert, handshake failure (552):
<= Recv SSL data, 2 bytes (0x2)
0000: .(
== Info: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
== Info: Closing connection 0
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0

@thiagowfx thiagowfx mentioned this issue Oct 30, 2023
Closed
@thatsIch thatsIch mentioned this issue Nov 7, 2023
Closed
@MatMercer
Copy link

Got:

57.65 npm ERR!     code: 'ECONNRESET',
57.65 npm ERR!     path: null,
57.65 npm ERR!     host: 'edgedl.me.gvt1.com',
57.65 npm ERR!     port: 443,
57.65 npm ERR!     localAddress: undefined

For VIVO ISP at south Brazil, this is weird, when connected to a USA VPN it worked, not blocked anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@vekien @OrKoN @RopoMen @MatMercer @hjaske @pattrykks