Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 19 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXX.X] duration: 30.001410875s FAIL (2025-08-07 21:42:43 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.0014358s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [......................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 6.400822847s FAIL (2025-08-07 21:42:41 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 12h1m14.02027627s) [upload=12435]: Upload finished. err: context deadline exceeded Aug 7 21:41:00.056743511 (12h1m34.571846093s) [upload=12451]: Starting upload Aug 7 21:41:00.056834896 (12h1m34.571937476s) [upload=12451]: Done starting upload Aug 7 21:41:00.056947293 (12h1m34.572049861s) [upload=12451]: Uploading segments... Aug 7 21:41:00.057253002 (12h1m34.572355591s) [upload=12451]: Got next segment. Inline: false Aug 7 21:41:00.057314022 (12h1m34.572416588s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:00.057326500 (12h1m34.572429079s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:06.802724338 (12h1m41.317826923s) [seg_pos={0 0}, upload=12451]: Begin upload segment... Aug 7 21:41:06.802779366 (12h1m41.317881933s) [seg_pos={0 0}, upload=12451]: Done begin upload segment. Aug 7 21:41:06.802802866 (12h1m41.317905445s) [noise=true , address=77.90.31.14:4663 , piece_id=MCP22S4S2267OULHW5SYQ6O7I3LG4PKWIMEQ45MS2KA4JOAHFY6A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.802894905 (12h1m41.317997502s) [noise=true , address=77.90.31.13:4627 , piece_id=FHLWRN7K4OU6HDQQ6X44TMLI7GPQPDJ4INQGGKMMCYSCSYGPR33Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803391882 (12h1m41.318494468s) [noise=true , address=104.36.82.10:4078 , piece_id=7JWKJPKOUISJF2EXKBWXMPCULDIUJRKZUVA72FOLY32PHJFVRD2Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803522360 (12h1m41.31862495s) [noise=true , address=104.36.82.15:4258 , piece_id=NF3KM2RU3RLV45BOPW46NW3UGL32WH473ADDXFROB2IQC3I54XGA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803904747 (12h1m41.31900733s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSU62T7KZCWTJ64CSFXAOMTFB3DY7QU67JWANKCHA64EYGJPU2GQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804010771 (12h1m41.319113359s) [noise=true , address=77.90.31.11:4553 , piece_id=Y56CRBHKAGCWU2PEKQIMAMGZ6TNMAWCSHIMCOQUEK7MK64EVWE4A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804197608 (12h1m41.319300187s) [noise=true , address=104.36.82.6:4875 , piece_id=5ZGFKU7IVKNZWIIW3A2E6S6ZVQMDHOYDUKGRWFL3YEMK3ZI47DEQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804479589 (12h1m41.319582176s) [noise=true , address=104.36.82.13:4186 , piece_id=P7Y3FZQF2T5Y6CRLKY22OOYZPJRUECW7VEYNCPMZCCWPH2OM47OQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804499574 (12h1m41.31960214s) [noise=true , address=77.90.31.21:4878 , piece_id=LVQCIDDIOFQA2RKX7HS5SRUBOWO7IIMD7IVQF7Y5OI3T3LDUGL2A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804953768 (12h1m41.320056351s) [noise=true , address=45.91.116.7:4009 , piece_id=OZXLMKK7LWVAAPIUI6WZ6WLYKDNKXA4T5PJG34JNUEU3A7ALAG3Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805026844 (12h1m41.320129434s) [noise=true , address=104.36.82.3:4767 , piece_id=GB7R5UALDIVEKCEFGWJMY4JJUPPNGSGUUJIQZYBHUKXRY4DCG3VA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805236078 (12h1m41.320338659s) [noise=true , address=45.91.116.6:4974 , piece_id=CSR4RAGCHGZ3AHYDZLQT4OFW7TLI5IEJHWD5Z6IGRG5ZKQGU3UFA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805491522 (12h1m41.320594111s) [noise=true , address=45.91.116.12:4227 , piece_id=5J6UANJSVQRV3A3HMPM6FETPYHZKFV3OLL7DLCV6WY3WENLTTGTA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805532740 (12h1m41.320635319s) [noise=true , address=45.91.116.13:4155 , piece_id=SVJJFQA5ZMIMHVT7E5JTF2S7PONZWD6S6T7WUYS63UH6MTOFNDZQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805966784 (12h1m41.321069381s) [noise=true , address=104.36.82.9:4005 , piece_id=HTJDYQJBCBGMVUWCEHH6CW5PVXRUAYAEYU42QYL4ZINXLWP66KQA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.806546714 (12h1m41.321649292s) [noise=true , address=45.91.116.5:4298 , piece_id=JWYNJUQ3MQFCC5W4L7QBXU2JR6OJQH5GPML6SMNUZWNO2LLYK7LA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.926689944 (12h1m41.441792523s) [upload=12451]: Next returned nil segment Aug 7 21:41:06.926709145 (12h1m41.441811733s) [upload=12451]: Waiting for error group managing segments... Aug 7 21:41:06.931954878 (12h1m41.447057458s) [noise=true , address=45.91.116.6:4974 , piece_id=CSR4RAGCHGZ3AHYDZLQT4OFW7TLI5IEJHWD5Z6IGRG5ZKQGU3UFA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.933221632 (12h1m41.448324213s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSU62T7KZCWTJ64CSFXAOMTFB3DY7QU67JWANKCHA64EYGJPU2GQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.933245521 (12h1m41.448348097s) [noise=true , address=45.91.116.12:4227 , piece_id=5J6UANJSVQRV3A3HMPM6FETPYHZKFV3OLL7DLCV6WY3WENLTTGTA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.935433763 (12h1m41.450536353s) [noise=true , address=77.90.31.14:4663 , piece_id=MCP22S4S2267OULHW5SYQ6O7I3LG4PKWIMEQ45MS2KA4JOAHFY6A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.935767291 (12h1m41.450869868s) [noise=true , address=45.91.116.13:4155 , piece_id=SVJJFQA5ZMIMHVT7E5JTF2S7PONZWD6S6T7WUYS63UH6MTOFNDZQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.938400107 (12h1m41.453502685s) [noise=true , address=104.36.82.13:4186 , piece_id=P7Y3FZQF2T5Y6CRLKY22OOYZPJRUECW7VEYNCPMZCCWPH2OM47OQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.938687716 (12h1m41.453790278s) [noise=true , address=45.91.116.5:4298 , piece_id=JWYNJUQ3MQFCC5W4L7QBXU2JR6OJQH5GPML6SMNUZWNO2LLYK7LA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.939447304 (12h1m41.454549875s) [noise=true , address=77.90.31.21:4878 , piece_id=LVQCIDDIOFQA2RKX7HS5SRUBOWO7IIMD7IVQF7Y5OI3T3LDUGL2A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.939805604 (12h1m41.454908182s) [noise=true , address=104.36.82.6:4875 , piece_id=5ZGFKU7IVKNZWIIW3A2E6S6ZVQMDHOYDUKGRWFL3YEMK3ZI47DEQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.950100367 (12h1m41.465202944s) [noise=true , address=45.91.116.7:4009 , piece_id=OZXLMKK7LWVAAPIUI6WZ6WLYKDNKXA4T5PJG34JNUEU3A7ALAG3Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.950539445 (12h1m41.465642035s) [noise=true , address=77.90.31.13:4627 , piece_id=FHLWRN7K4OU6HDQQ6X44TMLI7GPQPDJ4INQGGKMMCYSCSYGPR33Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953496508 (12h1m41.468599086s) [noise=true , address=77.90.31.11:4553 , piece_id=Y56CRBHKAGCWU2PEKQIMAMGZ6TNMAWCSHIMCOQUEK7MK64EVWE4A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953532645 (12h1m41.468635227s) [noise=true , address=104.36.82.10:4078 , piece_id=7JWKJPKOUISJF2EXKBWXMPCULDIUJRKZUVA72FOLY32PHJFVRD2Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953548888 (12h1m41.468651465s) [seg_pos={0 0}, upload=12451]: Segment reached optimal threshold of 13 pieces. Aug 7 21:41:06.953680391 (12h1m41.468782969s) [noise=true , address=104.36.82.15:4258 , piece_id=NF3KM2RU3RLV45BOPW46NW3UGL32WH473ADDXFROB2IQC3I54XGA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:06.953699436 (12h1m41.468802002s) [noise=true , address=104.36.82.3:4767 , piece_id=GB7R5UALDIVEKCEFGWJMY4JJUPPNGSGUUJIQZYBHUKXRY4DCG3VA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:06.953979656 (12h1m41.46908222s) [noise=true , address=104.36.82.9:4005 , piece_id=HTJDYQJBCBGMVUWCEHH6CW5PVXRUAYAEYU42QYL4ZINXLWP66KQA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: read tcp 159.223.143.39:53626->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:41:06.953998020 (12h1m41.469100586s) [seg_pos={0 0}, upload=12451]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:41:06.954079225 (12h1m41.469181809s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:06.954085123 (12h1m41.46918771s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:31.504450541 (12h2m6.019553121s) [upload=12451]: Done uploading segments... Aug 7 21:41:31.504481231 (12h2m6.019583808s) [upload=12451]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:31.625936603 (12h2m6.141039168s) [upload=12459]: Starting upload Aug 7 21:41:31.626009741 (12h2m6.141112327s) [upload=12459]: Done starting upload Aug 7 21:41:31.626253438 (12h2m6.141356016s) [upload=12459]: Uploading segments... Aug 7 21:41:31.626287931 (12h2m6.141390513s) [upload=12459]: Got next segment. Inline: false Aug 7 21:41:31.626319906 (12h2m6.141422482s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:31.626323686 (12h2m6.141426262s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:01.589273243 (12h2m36.104375832s) [upload=12459]: Next segment err: context deadline exceeded Aug 7 21:42:01.589453960 (12h2m36.104556539s) [upload=12459]: Done uploading segments... Aug 7 21:42:01.589475808 (12h2m36.104578383s) [upload=12459]: Upload finished. err: context deadline exceeded Duration: 6.400847378s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.........XX...........] duration: 40.954403ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:42:41 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 40.982608ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 226.786883ms FAIL (2025-08-07 21:42:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 226.833286ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [..................................XXX........XX..XX.........] duration: 46.032646ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:42:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 46.059355ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.........XX..........] duration: 30.079309ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:42:21 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 30.128866ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 679.329059ms FAIL (2025-08-07 21:42:20 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 679.35294ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 35 out of 60 is failed (threshold=3) [.......................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.001172535s FAIL (2025-08-07 21:42:14 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001223596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXX.] duration: 1.431903375s FAIL (2025-08-07 21:42:10 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g piece... Aug 7 21:39:36.313048033 (12h0m10.828150614s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313476014 (12h0m10.828578594s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313843993 (12h0m10.828946576s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314216946 (12h0m10.829319627s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314621829 (12h0m10.829724415s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314962361 (12h0m10.830064945s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315341892 (12h0m10.830444458s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315717142 (12h0m10.830819709s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316034230 (12h0m10.83113681s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316387250 (12h0m10.831489818s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316800121 (12h0m10.831902698s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317091487 (12h0m10.832194057s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317378915 (12h0m10.832481482s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317693529 (12h0m10.832796116s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318011241 (12h0m10.833113823s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318314555 (12h0m10.833417131s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.405758856 (12h0m10.920861443s) [upload=12426]: Next returned nil segment Aug 7 21:39:36.405776361 (12h0m10.920878951s) [upload=12426]: Waiting for error group managing segments... Aug 7 21:39:36.423798896 (12h0m10.938901465s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.427938435 (12h0m10.943041015s) [noise=true , address=45.91.116.10:4092 , piece_id=GN66X2MISCTHDBGGDWOMPAD44VZHKZRFOHNGYP3CEVKHTQMEVS5Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.438590001 (12h0m10.953692572s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.439652545 (12h0m10.954755226s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.445868473 (12h0m10.960971037s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.446471740 (12h0m10.961574309s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.447690219 (12h0m10.96279279s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.449661572 (12h0m10.964764152s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.450416189 (12h0m10.965518772s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.453686307 (12h0m10.968788884s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.466878370 (12h0m10.981980947s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.543317192 (12h0m11.058419762s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.551730838 (12h0m11.066833415s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606002356 (12h0m11.121104933s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606032771 (12h0m11.121135347s) [seg_pos={0 0}, upload=12426]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:36.606628860 (12h0m11.121731425s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606708385 (12h0m11.121810956s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606769362 (12h0m11.121871933s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bgD3PwFrAz1UmQvd8V2rNAf7dFMbjrUEvn1ATTpSM85HijKHS): protocol: read tcp 159.223.143.39:49096->104.36.82.43:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:36.606779304 (12h0m11.121881873s) [noise=true , address=104.36.82.8:4004 , piece_id=DSETT3WILT5UL3OR6W27WV24QD2MOIYG4IN5CMNCQ2QJE26AMLAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection Aug 7 21:39:36.606819004 (12h0m11.121921573s) [seg_pos={0 0}, upload=12426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:36.606946009 (12h0m11.122048586s) [upload=12426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:36.606959527 (12h0m11.122062104s) [upload=12426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:08.632421584 (12h1m43.147524172s) [upload=12426]: Done uploading segments... Aug 7 21:41:08.632450790 (12h1m43.147553358s) [upload=12426]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:08.750931310 (12h1m43.266033893s) [upload=12453]: Starting upload Aug 7 21:41:08.751044034 (12h1m43.266146621s) [upload=12453]: Done starting upload Aug 7 21:41:08.751168536 (12h1m43.2662711s) [upload=12453]: Uploading segments... Aug 7 21:41:08.751366735 (12h1m43.266469311s) [upload=12453]: Got next segment. Inline: false Aug 7 21:41:08.751423163 (12h1m43.266525746s) [upload=12453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:08.751431978 (12h1m43.26653456s) [upload=12453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:08.718711666 (12h2m43.233814253s) [upload=12453]: Next segment err: context deadline exceeded Aug 7 21:42:08.718867335 (12h2m43.233969903s) [upload=12453]: Done uploading segments... Aug 7 21:42:08.718888660 (12h2m43.233991231s) [upload=12453]: Upload finished. err: context deadline exceeded Duration: 1.432017637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.001329546s FAIL (2025-08-07 21:42:08 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g piece... Aug 7 21:39:36.313048033 (12h0m10.828150614s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313476014 (12h0m10.828578594s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313843993 (12h0m10.828946576s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314216946 (12h0m10.829319627s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314621829 (12h0m10.829724415s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314962361 (12h0m10.830064945s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315341892 (12h0m10.830444458s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315717142 (12h0m10.830819709s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316034230 (12h0m10.83113681s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316387250 (12h0m10.831489818s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316800121 (12h0m10.831902698s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317091487 (12h0m10.832194057s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317378915 (12h0m10.832481482s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317693529 (12h0m10.832796116s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318011241 (12h0m10.833113823s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318314555 (12h0m10.833417131s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.405758856 (12h0m10.920861443s) [upload=12426]: Next returned nil segment Aug 7 21:39:36.405776361 (12h0m10.920878951s) [upload=12426]: Waiting for error group managing segments... Aug 7 21:39:36.423798896 (12h0m10.938901465s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.427938435 (12h0m10.943041015s) [noise=true , address=45.91.116.10:4092 , piece_id=GN66X2MISCTHDBGGDWOMPAD44VZHKZRFOHNGYP3CEVKHTQMEVS5Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.438590001 (12h0m10.953692572s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.439652545 (12h0m10.954755226s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.445868473 (12h0m10.960971037s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.446471740 (12h0m10.961574309s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.447690219 (12h0m10.96279279s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.449661572 (12h0m10.964764152s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.450416189 (12h0m10.965518772s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.453686307 (12h0m10.968788884s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.466878370 (12h0m10.981980947s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.543317192 (12h0m11.058419762s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.551730838 (12h0m11.066833415s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606002356 (12h0m11.121104933s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606032771 (12h0m11.121135347s) [seg_pos={0 0}, upload=12426]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:36.606628860 (12h0m11.121731425s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606708385 (12h0m11.121810956s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606769362 (12h0m11.121871933s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bgD3PwFrAz1UmQvd8V2rNAf7dFMbjrUEvn1ATTpSM85HijKHS): protocol: read tcp 159.223.143.39:49096->104.36.82.43:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:36.606779304 (12h0m11.121881873s) [noise=true , address=104.36.82.8:4004 , piece_id=DSETT3WILT5UL3OR6W27WV24QD2MOIYG4IN5CMNCQ2QJE26AMLAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection Aug 7 21:39:36.606819004 (12h0m11.121921573s) [seg_pos={0 0}, upload=12426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:36.606946009 (12h0m11.122048586s) [upload=12426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:36.606959527 (12h0m11.122062104s) [upload=12426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:08.632421584 (12h1m43.147524172s) [upload=12426]: Done uploading segments... Aug 7 21:41:08.632450790 (12h1m43.147553358s) [upload=12426]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:08.750931310 (12h1m43.266033893s) [upload=12453]: Starting upload Aug 7 21:41:08.751044034 (12h1m43.266146621s) [upload=12453]: Done starting upload Aug 7 21:41:08.751168536 (12h1m43.2662711s) [upload=12453]: Uploading segments... Aug 7 21:41:08.751366735 (12h1m43.266469311s) [upload=12453]: Got next segment. Inline: false Aug 7 21:41:08.751423163 (12h1m43.266525746s) [upload=12453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:08.751431978 (12h1m43.26653456s) [upload=12453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:08.718711666 (12h2m43.233814253s) [upload=12453]: Next segment err: context deadline exceeded Aug 7 21:42:08.718867335 (12h2m43.233969903s) [upload=12453]: Done uploading segments... Aug 7 21:42:08.718888660 (12h2m43.233991231s) [upload=12453]: Upload finished. err: context deadline exceeded Duration: 1m0.001343218s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [...................................XXX........XX..XX........] duration: 85.621005ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:42:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 85.645954ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 1m0.000870841s FAIL (2025-08-07 21:42:03 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000896087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000643164s FAIL (2025-08-07 21:42:02 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 40 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:51.494104017 (11h56m26.009206585s) [upload=12372]: Starting upload Aug 7 21:35:51.494158922 (11h56m26.009261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000683353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.......................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000660663s FAIL (2025-08-07 21:42:01 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 12h1m14.02027627s) [upload=12435]: Upload finished. err: context deadline exceeded Aug 7 21:41:00.056743511 (12h1m34.571846093s) [upload=12451]: Starting upload Aug 7 21:41:00.056834896 (12h1m34.571937476s) [upload=12451]: Done starting upload Aug 7 21:41:00.056947293 (12h1m34.572049861s) [upload=12451]: Uploading segments... Aug 7 21:41:00.057253002 (12h1m34.572355591s) [upload=12451]: Got next segment. Inline: false Aug 7 21:41:00.057314022 (12h1m34.572416588s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:00.057326500 (12h1m34.572429079s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:06.802724338 (12h1m41.317826923s) [seg_pos={0 0}, upload=12451]: Begin upload segment... Aug 7 21:41:06.802779366 (12h1m41.317881933s) [seg_pos={0 0}, upload=12451]: Done begin upload segment. Aug 7 21:41:06.802802866 (12h1m41.317905445s) [noise=true , address=77.90.31.14:4663 , piece_id=MCP22S4S2267OULHW5SYQ6O7I3LG4PKWIMEQ45MS2KA4JOAHFY6A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.802894905 (12h1m41.317997502s) [noise=true , address=77.90.31.13:4627 , piece_id=FHLWRN7K4OU6HDQQ6X44TMLI7GPQPDJ4INQGGKMMCYSCSYGPR33Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803391882 (12h1m41.318494468s) [noise=true , address=104.36.82.10:4078 , piece_id=7JWKJPKOUISJF2EXKBWXMPCULDIUJRKZUVA72FOLY32PHJFVRD2Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803522360 (12h1m41.31862495s) [noise=true , address=104.36.82.15:4258 , piece_id=NF3KM2RU3RLV45BOPW46NW3UGL32WH473ADDXFROB2IQC3I54XGA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803904747 (12h1m41.31900733s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSU62T7KZCWTJ64CSFXAOMTFB3DY7QU67JWANKCHA64EYGJPU2GQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804010771 (12h1m41.319113359s) [noise=true , address=77.90.31.11:4553 , piece_id=Y56CRBHKAGCWU2PEKQIMAMGZ6TNMAWCSHIMCOQUEK7MK64EVWE4A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804197608 (12h1m41.319300187s) [noise=true , address=104.36.82.6:4875 , piece_id=5ZGFKU7IVKNZWIIW3A2E6S6ZVQMDHOYDUKGRWFL3YEMK3ZI47DEQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804479589 (12h1m41.319582176s) [noise=true , address=104.36.82.13:4186 , piece_id=P7Y3FZQF2T5Y6CRLKY22OOYZPJRUECW7VEYNCPMZCCWPH2OM47OQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804499574 (12h1m41.31960214s) [noise=true , address=77.90.31.21:4878 , piece_id=LVQCIDDIOFQA2RKX7HS5SRUBOWO7IIMD7IVQF7Y5OI3T3LDUGL2A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804953768 (12h1m41.320056351s) [noise=true , address=45.91.116.7:4009 , piece_id=OZXLMKK7LWVAAPIUI6WZ6WLYKDNKXA4T5PJG34JNUEU3A7ALAG3Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805026844 (12h1m41.320129434s) [noise=true , address=104.36.82.3:4767 , piece_id=GB7R5UALDIVEKCEFGWJMY4JJUPPNGSGUUJIQZYBHUKXRY4DCG3VA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805236078 (12h1m41.320338659s) [noise=true , address=45.91.116.6:4974 , piece_id=CSR4RAGCHGZ3AHYDZLQT4OFW7TLI5IEJHWD5Z6IGRG5ZKQGU3UFA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805491522 (12h1m41.320594111s) [noise=true , address=45.91.116.12:4227 , piece_id=5J6UANJSVQRV3A3HMPM6FETPYHZKFV3OLL7DLCV6WY3WENLTTGTA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805532740 (12h1m41.320635319s) [noise=true , address=45.91.116.13:4155 , piece_id=SVJJFQA5ZMIMHVT7E5JTF2S7PONZWD6S6T7WUYS63UH6MTOFNDZQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805966784 (12h1m41.321069381s) [noise=true , address=104.36.82.9:4005 , piece_id=HTJDYQJBCBGMVUWCEHH6CW5PVXRUAYAEYU42QYL4ZINXLWP66KQA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.806546714 (12h1m41.321649292s) [noise=true , address=45.91.116.5:4298 , piece_id=JWYNJUQ3MQFCC5W4L7QBXU2JR6OJQH5GPML6SMNUZWNO2LLYK7LA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.926689944 (12h1m41.441792523s) [upload=12451]: Next returned nil segment Aug 7 21:41:06.926709145 (12h1m41.441811733s) [upload=12451]: Waiting for error group managing segments... Aug 7 21:41:06.931954878 (12h1m41.447057458s) [noise=true , address=45.91.116.6:4974 , piece_id=CSR4RAGCHGZ3AHYDZLQT4OFW7TLI5IEJHWD5Z6IGRG5ZKQGU3UFA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.933221632 (12h1m41.448324213s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSU62T7KZCWTJ64CSFXAOMTFB3DY7QU67JWANKCHA64EYGJPU2GQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.933245521 (12h1m41.448348097s) [noise=true , address=45.91.116.12:4227 , piece_id=5J6UANJSVQRV3A3HMPM6FETPYHZKFV3OLL7DLCV6WY3WENLTTGTA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.935433763 (12h1m41.450536353s) [noise=true , address=77.90.31.14:4663 , piece_id=MCP22S4S2267OULHW5SYQ6O7I3LG4PKWIMEQ45MS2KA4JOAHFY6A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.935767291 (12h1m41.450869868s) [noise=true , address=45.91.116.13:4155 , piece_id=SVJJFQA5ZMIMHVT7E5JTF2S7PONZWD6S6T7WUYS63UH6MTOFNDZQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.938400107 (12h1m41.453502685s) [noise=true , address=104.36.82.13:4186 , piece_id=P7Y3FZQF2T5Y6CRLKY22OOYZPJRUECW7VEYNCPMZCCWPH2OM47OQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.938687716 (12h1m41.453790278s) [noise=true , address=45.91.116.5:4298 , piece_id=JWYNJUQ3MQFCC5W4L7QBXU2JR6OJQH5GPML6SMNUZWNO2LLYK7LA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.939447304 (12h1m41.454549875s) [noise=true , address=77.90.31.21:4878 , piece_id=LVQCIDDIOFQA2RKX7HS5SRUBOWO7IIMD7IVQF7Y5OI3T3LDUGL2A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.939805604 (12h1m41.454908182s) [noise=true , address=104.36.82.6:4875 , piece_id=5ZGFKU7IVKNZWIIW3A2E6S6ZVQMDHOYDUKGRWFL3YEMK3ZI47DEQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.950100367 (12h1m41.465202944s) [noise=true , address=45.91.116.7:4009 , piece_id=OZXLMKK7LWVAAPIUI6WZ6WLYKDNKXA4T5PJG34JNUEU3A7ALAG3Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.950539445 (12h1m41.465642035s) [noise=true , address=77.90.31.13:4627 , piece_id=FHLWRN7K4OU6HDQQ6X44TMLI7GPQPDJ4INQGGKMMCYSCSYGPR33Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953496508 (12h1m41.468599086s) [noise=true , address=77.90.31.11:4553 , piece_id=Y56CRBHKAGCWU2PEKQIMAMGZ6TNMAWCSHIMCOQUEK7MK64EVWE4A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953532645 (12h1m41.468635227s) [noise=true , address=104.36.82.10:4078 , piece_id=7JWKJPKOUISJF2EXKBWXMPCULDIUJRKZUVA72FOLY32PHJFVRD2Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953548888 (12h1m41.468651465s) [seg_pos={0 0}, upload=12451]: Segment reached optimal threshold of 13 pieces. Aug 7 21:41:06.953680391 (12h1m41.468782969s) [noise=true , address=104.36.82.15:4258 , piece_id=NF3KM2RU3RLV45BOPW46NW3UGL32WH473ADDXFROB2IQC3I54XGA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:06.953699436 (12h1m41.468802002s) [noise=true , address=104.36.82.3:4767 , piece_id=GB7R5UALDIVEKCEFGWJMY4JJUPPNGSGUUJIQZYBHUKXRY4DCG3VA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:06.953979656 (12h1m41.46908222s) [noise=true , address=104.36.82.9:4005 , piece_id=HTJDYQJBCBGMVUWCEHH6CW5PVXRUAYAEYU42QYL4ZINXLWP66KQA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: read tcp 159.223.143.39:53626->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:41:06.953998020 (12h1m41.469100586s) [seg_pos={0 0}, upload=12451]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:41:06.954079225 (12h1m41.469181809s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:06.954085123 (12h1m41.46918771s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:31.504450541 (12h2m6.019553121s) [upload=12451]: Done uploading segments... Aug 7 21:41:31.504481231 (12h2m6.019583808s) [upload=12451]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:31.625936603 (12h2m6.141039168s) [upload=12459]: Starting upload Aug 7 21:41:31.626009741 (12h2m6.141112327s) [upload=12459]: Done starting upload Aug 7 21:41:31.626253438 (12h2m6.141356016s) [upload=12459]: Uploading segments... Aug 7 21:41:31.626287931 (12h2m6.141390513s) [upload=12459]: Got next segment. Inline: false Aug 7 21:41:31.626319906 (12h2m6.141422482s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:31.626323686 (12h2m6.141426262s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:01.589273243 (12h2m36.104375832s) [upload=12459]: Next segment err: context deadline exceeded Aug 7 21:42:01.589453960 (12h2m36.104556539s) [upload=12459]: Done uploading segments... Aug 7 21:42:01.589475808 (12h2m36.104578383s) [upload=12459]: Upload finished. err: context deadline exceeded Duration: 30.000679008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 3.839141792s FAIL (2025-08-07 21:41:57 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 3.83916804s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [....................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00084079s FAIL (2025-08-07 21:41:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000863865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 34 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.00129801s FAIL (2025-08-07 21:41:43 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001316911s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 1.203132087s FAIL (2025-08-07 21:41:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.203151193s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 34 out of 60 is failed (threshold=3) [........................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 21.159462831s FAIL (2025-08-07 21:41:38 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Duration: 21.159489748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 447.458108ms FAIL (2025-08-07 21:41:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 447.483527ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 31.479857832s FAIL (2025-08-07 21:41:31 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43.934783297s) [upload=12428]: Upload finished. err: context deadline exceeded Aug 7 21:40:09.542009957 (12h0m44.057112542s) [upload=12435]: Starting upload Aug 7 21:40:09.542102758 (12h0m44.057205346s) [upload=12435]: Done starting upload Aug 7 21:40:09.542185753 (12h0m44.057288326s) [upload=12435]: Uploading segments... Aug 7 21:40:09.542450582 (12h0m44.057553163s) [upload=12435]: Got next segment. Inline: false Aug 7 21:40:09.542493961 (12h0m44.057596539s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:09.542499849 (12h0m44.057602431s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:39.505008014 (12h1m14.02011058s) [upload=12435]: Next segment err: context deadline exceeded Aug 7 21:40:39.505157414 (12h1m14.020259992s) [upload=12435]: Done uploading segments... Aug 7 21:40:39.505173700 (12h1m14.02027627s) [upload=12435]: Upload finished. err: context deadline exceeded Aug 7 21:41:00.056743511 (12h1m34.571846093s) [upload=12451]: Starting upload Aug 7 21:41:00.056834896 (12h1m34.571937476s) [upload=12451]: Done starting upload Aug 7 21:41:00.056947293 (12h1m34.572049861s) [upload=12451]: Uploading segments... Aug 7 21:41:00.057253002 (12h1m34.572355591s) [upload=12451]: Got next segment. Inline: false Aug 7 21:41:00.057314022 (12h1m34.572416588s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:00.057326500 (12h1m34.572429079s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:06.802724338 (12h1m41.317826923s) [seg_pos={0 0}, upload=12451]: Begin upload segment... Aug 7 21:41:06.802779366 (12h1m41.317881933s) [seg_pos={0 0}, upload=12451]: Done begin upload segment. Aug 7 21:41:06.802802866 (12h1m41.317905445s) [noise=true , address=77.90.31.14:4663 , piece_id=MCP22S4S2267OULHW5SYQ6O7I3LG4PKWIMEQ45MS2KA4JOAHFY6A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.802894905 (12h1m41.317997502s) [noise=true , address=77.90.31.13:4627 , piece_id=FHLWRN7K4OU6HDQQ6X44TMLI7GPQPDJ4INQGGKMMCYSCSYGPR33Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803391882 (12h1m41.318494468s) [noise=true , address=104.36.82.10:4078 , piece_id=7JWKJPKOUISJF2EXKBWXMPCULDIUJRKZUVA72FOLY32PHJFVRD2Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803522360 (12h1m41.31862495s) [noise=true , address=104.36.82.15:4258 , piece_id=NF3KM2RU3RLV45BOPW46NW3UGL32WH473ADDXFROB2IQC3I54XGA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803904747 (12h1m41.31900733s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSU62T7KZCWTJ64CSFXAOMTFB3DY7QU67JWANKCHA64EYGJPU2GQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804010771 (12h1m41.319113359s) [noise=true , address=77.90.31.11:4553 , piece_id=Y56CRBHKAGCWU2PEKQIMAMGZ6TNMAWCSHIMCOQUEK7MK64EVWE4A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804197608 (12h1m41.319300187s) [noise=true , address=104.36.82.6:4875 , piece_id=5ZGFKU7IVKNZWIIW3A2E6S6ZVQMDHOYDUKGRWFL3YEMK3ZI47DEQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804479589 (12h1m41.319582176s) [noise=true , address=104.36.82.13:4186 , piece_id=P7Y3FZQF2T5Y6CRLKY22OOYZPJRUECW7VEYNCPMZCCWPH2OM47OQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804499574 (12h1m41.31960214s) [noise=true , address=77.90.31.21:4878 , piece_id=LVQCIDDIOFQA2RKX7HS5SRUBOWO7IIMD7IVQF7Y5OI3T3LDUGL2A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804953768 (12h1m41.320056351s) [noise=true , address=45.91.116.7:4009 , piece_id=OZXLMKK7LWVAAPIUI6WZ6WLYKDNKXA4T5PJG34JNUEU3A7ALAG3Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805026844 (12h1m41.320129434s) [noise=true , address=104.36.82.3:4767 , piece_id=GB7R5UALDIVEKCEFGWJMY4JJUPPNGSGUUJIQZYBHUKXRY4DCG3VA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805236078 (12h1m41.320338659s) [noise=true , address=45.91.116.6:4974 , piece_id=CSR4RAGCHGZ3AHYDZLQT4OFW7TLI5IEJHWD5Z6IGRG5ZKQGU3UFA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805491522 (12h1m41.320594111s) [noise=true , address=45.91.116.12:4227 , piece_id=5J6UANJSVQRV3A3HMPM6FETPYHZKFV3OLL7DLCV6WY3WENLTTGTA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805532740 (12h1m41.320635319s) [noise=true , address=45.91.116.13:4155 , piece_id=SVJJFQA5ZMIMHVT7E5JTF2S7PONZWD6S6T7WUYS63UH6MTOFNDZQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805966784 (12h1m41.321069381s) [noise=true , address=104.36.82.9:4005 , piece_id=HTJDYQJBCBGMVUWCEHH6CW5PVXRUAYAEYU42QYL4ZINXLWP66KQA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.806546714 (12h1m41.321649292s) [noise=true , address=45.91.116.5:4298 , piece_id=JWYNJUQ3MQFCC5W4L7QBXU2JR6OJQH5GPML6SMNUZWNO2LLYK7LA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.926689944 (12h1m41.441792523s) [upload=12451]: Next returned nil segment Aug 7 21:41:06.926709145 (12h1m41.441811733s) [upload=12451]: Waiting for error group managing segments... Aug 7 21:41:06.931954878 (12h1m41.447057458s) [noise=true , address=45.91.116.6:4974 , piece_id=CSR4RAGCHGZ3AHYDZLQT4OFW7TLI5IEJHWD5Z6IGRG5ZKQGU3UFA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.933221632 (12h1m41.448324213s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSU62T7KZCWTJ64CSFXAOMTFB3DY7QU67JWANKCHA64EYGJPU2GQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.933245521 (12h1m41.448348097s) [noise=true , address=45.91.116.12:4227 , piece_id=5J6UANJSVQRV3A3HMPM6FETPYHZKFV3OLL7DLCV6WY3WENLTTGTA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.935433763 (12h1m41.450536353s) [noise=true , address=77.90.31.14:4663 , piece_id=MCP22S4S2267OULHW5SYQ6O7I3LG4PKWIMEQ45MS2KA4JOAHFY6A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.935767291 (12h1m41.450869868s) [noise=true , address=45.91.116.13:4155 , piece_id=SVJJFQA5ZMIMHVT7E5JTF2S7PONZWD6S6T7WUYS63UH6MTOFNDZQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.938400107 (12h1m41.453502685s) [noise=true , address=104.36.82.13:4186 , piece_id=P7Y3FZQF2T5Y6CRLKY22OOYZPJRUECW7VEYNCPMZCCWPH2OM47OQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.938687716 (12h1m41.453790278s) [noise=true , address=45.91.116.5:4298 , piece_id=JWYNJUQ3MQFCC5W4L7QBXU2JR6OJQH5GPML6SMNUZWNO2LLYK7LA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.939447304 (12h1m41.454549875s) [noise=true , address=77.90.31.21:4878 , piece_id=LVQCIDDIOFQA2RKX7HS5SRUBOWO7IIMD7IVQF7Y5OI3T3LDUGL2A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.939805604 (12h1m41.454908182s) [noise=true , address=104.36.82.6:4875 , piece_id=5ZGFKU7IVKNZWIIW3A2E6S6ZVQMDHOYDUKGRWFL3YEMK3ZI47DEQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.950100367 (12h1m41.465202944s) [noise=true , address=45.91.116.7:4009 , piece_id=OZXLMKK7LWVAAPIUI6WZ6WLYKDNKXA4T5PJG34JNUEU3A7ALAG3Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.950539445 (12h1m41.465642035s) [noise=true , address=77.90.31.13:4627 , piece_id=FHLWRN7K4OU6HDQQ6X44TMLI7GPQPDJ4INQGGKMMCYSCSYGPR33Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953496508 (12h1m41.468599086s) [noise=true , address=77.90.31.11:4553 , piece_id=Y56CRBHKAGCWU2PEKQIMAMGZ6TNMAWCSHIMCOQUEK7MK64EVWE4A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953532645 (12h1m41.468635227s) [noise=true , address=104.36.82.10:4078 , piece_id=7JWKJPKOUISJF2EXKBWXMPCULDIUJRKZUVA72FOLY32PHJFVRD2Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953548888 (12h1m41.468651465s) [seg_pos={0 0}, upload=12451]: Segment reached optimal threshold of 13 pieces. Aug 7 21:41:06.953680391 (12h1m41.468782969s) [noise=true , address=104.36.82.15:4258 , piece_id=NF3KM2RU3RLV45BOPW46NW3UGL32WH473ADDXFROB2IQC3I54XGA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:06.953699436 (12h1m41.468802002s) [noise=true , address=104.36.82.3:4767 , piece_id=GB7R5UALDIVEKCEFGWJMY4JJUPPNGSGUUJIQZYBHUKXRY4DCG3VA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:06.953979656 (12h1m41.46908222s) [noise=true , address=104.36.82.9:4005 , piece_id=HTJDYQJBCBGMVUWCEHH6CW5PVXRUAYAEYU42QYL4ZINXLWP66KQA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: read tcp 159.223.143.39:53626->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:41:06.953998020 (12h1m41.469100586s) [seg_pos={0 0}, upload=12451]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:41:06.954079225 (12h1m41.469181809s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:06.954085123 (12h1m41.46918771s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:31.504450541 (12h2m6.019553121s) [upload=12451]: Done uploading segments... Aug 7 21:41:31.504481231 (12h2m6.019583808s) [upload=12451]: Upload finished. err: metaclient: context deadline exceeded Duration: 31.47988334s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.........XX.........] duration: 77.326554ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:41:28 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 77.356866ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXX.] duration: 22.426453312s FAIL (2025-08-07 21:41:21 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 22.426481933s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000561609s FAIL (2025-08-07 21:41:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000596814s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 37 out of 60 is failed (threshold=3) [.....................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000437497s FAIL (2025-08-07 21:41:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000457075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 2.513332818s FAIL (2025-08-07 21:41:13 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 2.513372455s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 33 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 25.431117427s FAIL (2025-08-07 21:41:12 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 25.431136266s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m38.782279426s FAIL (2025-08-07 21:41:08 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=12426]: Got next segment. Inline: false Aug 7 21:39:29.880419194 (12h0m4.395521764s) [upload=12426]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:29.880423782 (12h0m4.395526358s) [upload=12426]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:36.311856712 (12h0m10.826959299s) [seg_pos={0 0}, upload=12426]: Begin upload segment... Aug 7 21:39:36.311922714 (12h0m10.827025397s) [seg_pos={0 0}, upload=12426]: Done begin upload segment. Aug 7 21:39:36.311958150 (12h0m10.827060737s) [noise=true , address=45.91.116.10:4092 , piece_id=GN66X2MISCTHDBGGDWOMPAD44VZHKZRFOHNGYP3CEVKHTQMEVS5Q, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.312501389 (12h0m10.827603982s) [noise=true , address=104.36.82.8:4004 , piece_id=DSETT3WILT5UL3OR6W27WV24QD2MOIYG4IN5CMNCQ2QJE26AMLAA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313048033 (12h0m10.828150614s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313476014 (12h0m10.828578594s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313843993 (12h0m10.828946576s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314216946 (12h0m10.829319627s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314621829 (12h0m10.829724415s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314962361 (12h0m10.830064945s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315341892 (12h0m10.830444458s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315717142 (12h0m10.830819709s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316034230 (12h0m10.83113681s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316387250 (12h0m10.831489818s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316800121 (12h0m10.831902698s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317091487 (12h0m10.832194057s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317378915 (12h0m10.832481482s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317693529 (12h0m10.832796116s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318011241 (12h0m10.833113823s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318314555 (12h0m10.833417131s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.405758856 (12h0m10.920861443s) [upload=12426]: Next returned nil segment Aug 7 21:39:36.405776361 (12h0m10.920878951s) [upload=12426]: Waiting for error group managing segments... Aug 7 21:39:36.423798896 (12h0m10.938901465s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.427938435 (12h0m10.943041015s) [noise=true , address=45.91.116.10:4092 , piece_id=GN66X2MISCTHDBGGDWOMPAD44VZHKZRFOHNGYP3CEVKHTQMEVS5Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.438590001 (12h0m10.953692572s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.439652545 (12h0m10.954755226s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.445868473 (12h0m10.960971037s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.446471740 (12h0m10.961574309s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.447690219 (12h0m10.96279279s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.449661572 (12h0m10.964764152s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.450416189 (12h0m10.965518772s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.453686307 (12h0m10.968788884s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.466878370 (12h0m10.981980947s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.543317192 (12h0m11.058419762s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.551730838 (12h0m11.066833415s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606002356 (12h0m11.121104933s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606032771 (12h0m11.121135347s) [seg_pos={0 0}, upload=12426]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:36.606628860 (12h0m11.121731425s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606708385 (12h0m11.121810956s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606769362 (12h0m11.121871933s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bgD3PwFrAz1UmQvd8V2rNAf7dFMbjrUEvn1ATTpSM85HijKHS): protocol: read tcp 159.223.143.39:49096->104.36.82.43:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:36.606779304 (12h0m11.121881873s) [noise=true , address=104.36.82.8:4004 , piece_id=DSETT3WILT5UL3OR6W27WV24QD2MOIYG4IN5CMNCQ2QJE26AMLAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection Aug 7 21:39:36.606819004 (12h0m11.121921573s) [seg_pos={0 0}, upload=12426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:36.606946009 (12h0m11.122048586s) [upload=12426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:36.606959527 (12h0m11.122062104s) [upload=12426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:08.632421584 (12h1m43.147524172s) [upload=12426]: Done uploading segments... Aug 7 21:41:08.632450790 (12h1m43.147553358s) [upload=12426]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m38.782311041s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [....................................XXX........XX..XX.......] duration: 14.331789521s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:41:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 14.331815401s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001120853s FAIL (2025-08-07 21:41:03 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00116273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000658514s FAIL (2025-08-07 21:40:56 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:51.494104017 (11h56m26.009206585s) [upload=12372]: Starting upload Aug 7 21:35:51.494158922 (11h56m26.009261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000680642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 34 out of 60 is failed (threshold=3) [.........................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001446365s FAIL (2025-08-07 21:40:48 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001488502s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [......................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.002372319s FAIL (2025-08-07 21:40:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002394812s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000885468s FAIL (2025-08-07 21:40:47 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000914787s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 260.413048ms FAIL (2025-08-07 21:40:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 260.439203ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000564099s FAIL (2025-08-07 21:40:39 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 1.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Aug 7 21:34:42.716392204 (11h55m17.23149477s) [upload=12355]: Starting upload Aug 7 21:34:42.716466463 (11h55m17.231569048s) [upload=12355]: Done starting upload Aug 7 21:34:42.716631833 (11h55m17.231734402s) [upload=12355]: Uploading segments... Aug 7 21:34:42.716905636 (11h55m17.232008212s) [upload=12355]: Got next segment. Inline: false Aug 7 21:34:42.716950835 (11h55m17.232053401s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:42.716955102 (11h55m17.232057682s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:12.678206111 (11h55m47.193308708s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:35:12.678280249 (11h55m47.193382812s) [upload=12355]: Done uploading segments... Aug 7 21:35:12.678299114 (11h55m47.193401676s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:35:28.227195655 (11h56m2.742298223s) [upload=12367]: Starting upload Aug 7 21:35:28.227255087 (11h56m2.742357657s) [upload=12367]: Done starting upload Aug 7 21:35:28.227336170 (11h56m2.742438737s) [upload=12367]: Uploading segments... Aug 7 21:35:28.227548036 (11h56m2.742650613s) [upload=12367]: Got next segment. Inline: false Aug 7 21:35:28.227579186 (11h56m2.742681753s) [upload=12367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:28.227583627 (11h56m2.742686203s) [upload=12367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:58.195736071 (11h56m32.710838658s) [upload=12367]: Next segment err: context deadline exceeded Aug 7 21:35:58.195810513 (11h56m32.710913091s) [upload=12367]: Done uploading segments... Aug 7 21:35:58.195829651 (11h56m32.710932225s) [upload=12367]: Upload finished. err: context deadline exceeded Aug 7 21:36:37.527994350 (11h57m12.043096926s) [upload=12382]: Starting upload Aug 7 21:36:37.528079972 (11h57m12.043182536s) [upload=12382]: Done starting upload Aug 7 21:36:37.528185695 (11h57m12.043288268s) [upload=12382]: Uploading segments... Aug 7 21:36:37.528463728 (11h57m12.043566306s) [upload=12382]: Got next segment. Inline: false Aug 7 21:36:37.528594446 (11h57m12.043697027s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:37.528601324 (11h57m12.043703906s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:07.486412463 (11h57m42.001515034s) [upload=12382]: Next segment err: context deadline exceeded Aug 7 21:37:07.486493636 (11h57m42.001596204s) [upload=12382]: Done uploading segments... Aug 7 21:37:07.486510871 (11h57m42.001613436s) [upload=12382]: Upload finished. err: context deadline exceeded Aug 7 21:37:34.078813217 (11h58m8.593915791s) [upload=12397]: Starting upload Aug 7 21:37:34.078881152 (11h58m8.59398373s) [upload=12397]: Done starting upload Aug 7 21:37:34.079073995 (11h58m8.594176557s) [upload=12397]: Uploading segments... Aug 7 21:37:34.080463095 (11h58m8.59556567s) [upload=12397]: Got next segment. Inline: false Aug 7 21:37:34.080601937 (11h58m8.595704517s) [upload=12397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.080608666 (11h58m8.595711243s) [upload=12397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:04.048079311 (11h58m38.563181877s) [upload=12397]: Next segment err: context deadline exceeded Aug 7 21:38:04.048155743 (11h58m38.56325832s) [upload=12397]: Done uploading segments... Aug 7 21:38:04.048174465 (11h58m38.563277044s) [upload=12397]: Upload finished. err: context deadline exceeded Aug 7 21:38:07.629401418 (11h58m42.144503984s) [upload=12406]: Starting upload Aug 7 21:38:07.629479055 (11h58m42.144581624s) [upload=12406]: Done starting upload Aug 7 21:38:07.629571988 (11h58m42.144674564s) [upload=12406]: Uploading segments... Aug 7 21:38:07.629900317 (11h58m42.145002894s) [upload=12406]: Got next segment. Inline: false Aug 7 21:38:07.629953043 (11h58m42.145055624s) [upload=12406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:07.629965392 (11h58m42.145067979s) [upload=12406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:37.597200052 (11h59m12.112302638s) [upload=12406]: Next segment err: context deadline exceeded Aug 7 21:38:37.597290448 (11h59m12.112393017s) [upload=12406]: Done uploading segments... Aug 7 21:38:37.597308037 (11h59m12.112410608s) [upload=12406]: Upload finished. err: context deadline exceeded Aug 7 21:39:09.025809202 (11h59m43.540911779s) [upload=12420]: Starting upload Aug 7 21:39:09.025863832 (11h59m43.540966396s) [upload=12420]: Done starting upload Aug 7 21:39:09.026099227 (11h59m43.541201795s) [upload=12420]: Uploading segments... Aug 7 21:39:09.026135578 (11h59m43.541238163s) [upload=12420]: Got next segment. Inline: false Aug 7 21:39:09.026165709 (11h59m43.541268285s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:09.026170416 (11h59m43.541272994s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:38.991391623 (12h0m13.50649421s) [upload=12420]: Next segment err: context deadline exceeded Aug 7 21:39:38.991416990 (12h0m13.506519578s) [upload=12420]: Done uploading segments... Aug 7 21:39:38.991464450 (12h0m13.506567027s) [upload=12420]: Upload finished. err: context deadline exceeded Aug 7 21:39:39.452827409 (12h0m13.967929986s) [upload=12428]: Starting upload Aug 7 21:39:39.452919406 (12h0m13.968021972s) [upload=12428]: Done starting upload Aug 7 21:39:39.453320471 (12h0m13.968423039s) [upload=12428]: Uploading segments... Aug 7 21:39:39.453387273 (12h0m13.968489855s) [upload=12428]: Got next segment. Inline: false Aug 7 21:39:39.453416478 (12h0m13.968519054s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:39.453420854 (12h0m13.96852343s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:09.419592862 (12h0m43.934695447s) [upload=12428]: Next segment err: context deadline exceeded Aug 7 21:40:09.419652288 (12h0m43.93475487s) [upload=12428]: Done uploading segments... Aug 7 21:40:09.419680730 (12h0m43.934783297s) [upload=12428]: Upload finished. err: context deadline exceeded Aug 7 21:40:09.542009957 (12h0m44.057112542s) [upload=12435]: Starting upload Aug 7 21:40:09.542102758 (12h0m44.057205346s) [upload=12435]: Done starting upload Aug 7 21:40:09.542185753 (12h0m44.057288326s) [upload=12435]: Uploading segments... Aug 7 21:40:09.542450582 (12h0m44.057553163s) [upload=12435]: Got next segment. Inline: false Aug 7 21:40:09.542493961 (12h0m44.057596539s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:09.542499849 (12h0m44.057602431s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:39.505008014 (12h1m14.02011058s) [upload=12435]: Next segment err: context deadline exceeded Aug 7 21:40:39.505157414 (12h1m14.020259992s) [upload=12435]: Done uploading segments... Aug 7 21:40:39.505173700 (12h1m14.02027627s) [upload=12435]: Upload finished. err: context deadline exceeded Duration: 30.000580084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.00106221s FAIL (2025-08-07 21:40:31 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00107597s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.204586597s FAIL (2025-08-07 21:40:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 1.204613881s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001816224s FAIL (2025-08-07 21:40:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001846077s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.........XX........] duration: 30.74089ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:40:16 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 30.758602ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001590241s FAIL (2025-08-07 21:40:09 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Aug 7 21:34:42.716392204 (11h55m17.23149477s) [upload=12355]: Starting upload Aug 7 21:34:42.716466463 (11h55m17.231569048s) [upload=12355]: Done starting upload Aug 7 21:34:42.716631833 (11h55m17.231734402s) [upload=12355]: Uploading segments... Aug 7 21:34:42.716905636 (11h55m17.232008212s) [upload=12355]: Got next segment. Inline: false Aug 7 21:34:42.716950835 (11h55m17.232053401s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:42.716955102 (11h55m17.232057682s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:12.678206111 (11h55m47.193308708s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:35:12.678280249 (11h55m47.193382812s) [upload=12355]: Done uploading segments... Aug 7 21:35:12.678299114 (11h55m47.193401676s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:35:28.227195655 (11h56m2.742298223s) [upload=12367]: Starting upload Aug 7 21:35:28.227255087 (11h56m2.742357657s) [upload=12367]: Done starting upload Aug 7 21:35:28.227336170 (11h56m2.742438737s) [upload=12367]: Uploading segments... Aug 7 21:35:28.227548036 (11h56m2.742650613s) [upload=12367]: Got next segment. Inline: false Aug 7 21:35:28.227579186 (11h56m2.742681753s) [upload=12367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:28.227583627 (11h56m2.742686203s) [upload=12367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:58.195736071 (11h56m32.710838658s) [upload=12367]: Next segment err: context deadline exceeded Aug 7 21:35:58.195810513 (11h56m32.710913091s) [upload=12367]: Done uploading segments... Aug 7 21:35:58.195829651 (11h56m32.710932225s) [upload=12367]: Upload finished. err: context deadline exceeded Aug 7 21:36:37.527994350 (11h57m12.043096926s) [upload=12382]: Starting upload Aug 7 21:36:37.528079972 (11h57m12.043182536s) [upload=12382]: Done starting upload Aug 7 21:36:37.528185695 (11h57m12.043288268s) [upload=12382]: Uploading segments... Aug 7 21:36:37.528463728 (11h57m12.043566306s) [upload=12382]: Got next segment. Inline: false Aug 7 21:36:37.528594446 (11h57m12.043697027s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:37.528601324 (11h57m12.043703906s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:07.486412463 (11h57m42.001515034s) [upload=12382]: Next segment err: context deadline exceeded Aug 7 21:37:07.486493636 (11h57m42.001596204s) [upload=12382]: Done uploading segments... Aug 7 21:37:07.486510871 (11h57m42.001613436s) [upload=12382]: Upload finished. err: context deadline exceeded Aug 7 21:37:34.078813217 (11h58m8.593915791s) [upload=12397]: Starting upload Aug 7 21:37:34.078881152 (11h58m8.59398373s) [upload=12397]: Done starting upload Aug 7 21:37:34.079073995 (11h58m8.594176557s) [upload=12397]: Uploading segments... Aug 7 21:37:34.080463095 (11h58m8.59556567s) [upload=12397]: Got next segment. Inline: false Aug 7 21:37:34.080601937 (11h58m8.595704517s) [upload=12397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.080608666 (11h58m8.595711243s) [upload=12397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:04.048079311 (11h58m38.563181877s) [upload=12397]: Next segment err: context deadline exceeded Aug 7 21:38:04.048155743 (11h58m38.56325832s) [upload=12397]: Done uploading segments... Aug 7 21:38:04.048174465 (11h58m38.563277044s) [upload=12397]: Upload finished. err: context deadline exceeded Aug 7 21:38:07.629401418 (11h58m42.144503984s) [upload=12406]: Starting upload Aug 7 21:38:07.629479055 (11h58m42.144581624s) [upload=12406]: Done starting upload Aug 7 21:38:07.629571988 (11h58m42.144674564s) [upload=12406]: Uploading segments... Aug 7 21:38:07.629900317 (11h58m42.145002894s) [upload=12406]: Got next segment. Inline: false Aug 7 21:38:07.629953043 (11h58m42.145055624s) [upload=12406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:07.629965392 (11h58m42.145067979s) [upload=12406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:37.597200052 (11h59m12.112302638s) [upload=12406]: Next segment err: context deadline exceeded Aug 7 21:38:37.597290448 (11h59m12.112393017s) [upload=12406]: Done uploading segments... Aug 7 21:38:37.597308037 (11h59m12.112410608s) [upload=12406]: Upload finished. err: context deadline exceeded Aug 7 21:39:09.025809202 (11h59m43.540911779s) [upload=12420]: Starting upload Aug 7 21:39:09.025863832 (11h59m43.540966396s) [upload=12420]: Done starting upload Aug 7 21:39:09.026099227 (11h59m43.541201795s) [upload=12420]: Uploading segments... Aug 7 21:39:09.026135578 (11h59m43.541238163s) [upload=12420]: Got next segment. Inline: false Aug 7 21:39:09.026165709 (11h59m43.541268285s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:09.026170416 (11h59m43.541272994s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:38.991391623 (12h0m13.50649421s) [upload=12420]: Next segment err: context deadline exceeded Aug 7 21:39:38.991416990 (12h0m13.506519578s) [upload=12420]: Done uploading segments... Aug 7 21:39:38.991464450 (12h0m13.506567027s) [upload=12420]: Upload finished. err: context deadline exceeded Aug 7 21:39:39.452827409 (12h0m13.967929986s) [upload=12428]: Starting upload Aug 7 21:39:39.452919406 (12h0m13.968021972s) [upload=12428]: Done starting upload Aug 7 21:39:39.453320471 (12h0m13.968423039s) [upload=12428]: Uploading segments... Aug 7 21:39:39.453387273 (12h0m13.968489855s) [upload=12428]: Got next segment. Inline: false Aug 7 21:39:39.453416478 (12h0m13.968519054s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:39.453420854 (12h0m13.96852343s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:09.419592862 (12h0m43.934695447s) [upload=12428]: Next segment err: context deadline exceeded Aug 7 21:40:09.419652288 (12h0m43.93475487s) [upload=12428]: Done uploading segments... Aug 7 21:40:09.419680730 (12h0m43.934783297s) [upload=12428]: Upload finished. err: context deadline exceeded Duration: 30.001614302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 9m5.470195367s FAIL (2025-08-07 21:40:01 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 9m5.470209648s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 33 out of 60 is failed (threshold=3) [..........................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000944004s FAIL (2025-08-07 21:39:56 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000965293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000781322s FAIL (2025-08-07 21:39:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 09s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:51.494104017 (11h56m26.009206585s) [upload=12372]: Starting upload Aug 7 21:35:51.494158922 (11h56m26.009261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000794308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00095659s FAIL (2025-08-07 21:39:51 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000969878s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.....................................XXX........XX..XX......] duration: 112.785718ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:39:49 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 112.811316ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000845992s FAIL (2025-08-07 21:39:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000860744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 35 out of 60 is failed (threshold=3) [.......................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000909236s FAIL (2025-08-07 21:39:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001059457s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000808226s FAIL (2025-08-07 21:39:38 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== =12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Aug 7 21:34:42.716392204 (11h55m17.23149477s) [upload=12355]: Starting upload Aug 7 21:34:42.716466463 (11h55m17.231569048s) [upload=12355]: Done starting upload Aug 7 21:34:42.716631833 (11h55m17.231734402s) [upload=12355]: Uploading segments... Aug 7 21:34:42.716905636 (11h55m17.232008212s) [upload=12355]: Got next segment. Inline: false Aug 7 21:34:42.716950835 (11h55m17.232053401s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:42.716955102 (11h55m17.232057682s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:12.678206111 (11h55m47.193308708s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:35:12.678280249 (11h55m47.193382812s) [upload=12355]: Done uploading segments... Aug 7 21:35:12.678299114 (11h55m47.193401676s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:35:28.227195655 (11h56m2.742298223s) [upload=12367]: Starting upload Aug 7 21:35:28.227255087 (11h56m2.742357657s) [upload=12367]: Done starting upload Aug 7 21:35:28.227336170 (11h56m2.742438737s) [upload=12367]: Uploading segments... Aug 7 21:35:28.227548036 (11h56m2.742650613s) [upload=12367]: Got next segment. Inline: false Aug 7 21:35:28.227579186 (11h56m2.742681753s) [upload=12367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:28.227583627 (11h56m2.742686203s) [upload=12367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:58.195736071 (11h56m32.710838658s) [upload=12367]: Next segment err: context deadline exceeded Aug 7 21:35:58.195810513 (11h56m32.710913091s) [upload=12367]: Done uploading segments... Aug 7 21:35:58.195829651 (11h56m32.710932225s) [upload=12367]: Upload finished. err: context deadline exceeded Aug 7 21:36:37.527994350 (11h57m12.043096926s) [upload=12382]: Starting upload Aug 7 21:36:37.528079972 (11h57m12.043182536s) [upload=12382]: Done starting upload Aug 7 21:36:37.528185695 (11h57m12.043288268s) [upload=12382]: Uploading segments... Aug 7 21:36:37.528463728 (11h57m12.043566306s) [upload=12382]: Got next segment. Inline: false Aug 7 21:36:37.528594446 (11h57m12.043697027s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:37.528601324 (11h57m12.043703906s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:07.486412463 (11h57m42.001515034s) [upload=12382]: Next segment err: context deadline exceeded Aug 7 21:37:07.486493636 (11h57m42.001596204s) [upload=12382]: Done uploading segments... Aug 7 21:37:07.486510871 (11h57m42.001613436s) [upload=12382]: Upload finished. err: context deadline exceeded Aug 7 21:37:34.078813217 (11h58m8.593915791s) [upload=12397]: Starting upload Aug 7 21:37:34.078881152 (11h58m8.59398373s) [upload=12397]: Done starting upload Aug 7 21:37:34.079073995 (11h58m8.594176557s) [upload=12397]: Uploading segments... Aug 7 21:37:34.080463095 (11h58m8.59556567s) [upload=12397]: Got next segment. Inline: false Aug 7 21:37:34.080601937 (11h58m8.595704517s) [upload=12397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.080608666 (11h58m8.595711243s) [upload=12397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:04.048079311 (11h58m38.563181877s) [upload=12397]: Next segment err: context deadline exceeded Aug 7 21:38:04.048155743 (11h58m38.56325832s) [upload=12397]: Done uploading segments... Aug 7 21:38:04.048174465 (11h58m38.563277044s) [upload=12397]: Upload finished. err: context deadline exceeded Aug 7 21:38:07.629401418 (11h58m42.144503984s) [upload=12406]: Starting upload Aug 7 21:38:07.629479055 (11h58m42.144581624s) [upload=12406]: Done starting upload Aug 7 21:38:07.629571988 (11h58m42.144674564s) [upload=12406]: Uploading segments... Aug 7 21:38:07.629900317 (11h58m42.145002894s) [upload=12406]: Got next segment. Inline: false Aug 7 21:38:07.629953043 (11h58m42.145055624s) [upload=12406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:07.629965392 (11h58m42.145067979s) [upload=12406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:37.597200052 (11h59m12.112302638s) [upload=12406]: Next segment err: context deadline exceeded Aug 7 21:38:37.597290448 (11h59m12.112393017s) [upload=12406]: Done uploading segments... Aug 7 21:38:37.597308037 (11h59m12.112410608s) [upload=12406]: Upload finished. err: context deadline exceeded Aug 7 21:39:09.025809202 (11h59m43.540911779s) [upload=12420]: Starting upload Aug 7 21:39:09.025863832 (11h59m43.540966396s) [upload=12420]: Done starting upload Aug 7 21:39:09.026099227 (11h59m43.541201795s) [upload=12420]: Uploading segments... Aug 7 21:39:09.026135578 (11h59m43.541238163s) [upload=12420]: Got next segment. Inline: false Aug 7 21:39:09.026165709 (11h59m43.541268285s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:09.026170416 (11h59m43.541272994s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:38.991391623 (12h0m13.50649421s) [upload=12420]: Next segment err: context deadline exceeded Aug 7 21:39:38.991416990 (12h0m13.506519578s) [upload=12420]: Done uploading segments... Aug 7 21:39:38.991464450 (12h0m13.506567027s) [upload=12420]: Upload finished. err: context deadline exceeded Duration: 30.000829624s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000968043s FAIL (2025-08-07 21:39:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001059043s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)