Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 919.989513ms FAIL (2025-08-07 21:45:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 920.017533ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [..............................XXX........XX..XX.............] duration: 61.506662ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:45:46 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 61.530727ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....] duration: 503.362494ms FAIL (2025-08-07 21:45:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.381767ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...] duration: 1.926195839s FAIL (2025-08-07 21:45:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.926216833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 231.36061ms FAIL (2025-08-07 21:45:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 231.402156ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.365554273s FAIL (2025-08-07 21:45:20 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 1.365598213s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X..] duration: 286.679918ms FAIL (2025-08-07 21:45:18 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 286.71144ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXX.....] duration: 1.441086014s FAIL (2025-08-07 21:45:17 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g piece... Aug 7 21:39:36.313048033 (12h0m10.828150614s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313476014 (12h0m10.828578594s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313843993 (12h0m10.828946576s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314216946 (12h0m10.829319627s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314621829 (12h0m10.829724415s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314962361 (12h0m10.830064945s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315341892 (12h0m10.830444458s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315717142 (12h0m10.830819709s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316034230 (12h0m10.83113681s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316387250 (12h0m10.831489818s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316800121 (12h0m10.831902698s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317091487 (12h0m10.832194057s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317378915 (12h0m10.832481482s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317693529 (12h0m10.832796116s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318011241 (12h0m10.833113823s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318314555 (12h0m10.833417131s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.405758856 (12h0m10.920861443s) [upload=12426]: Next returned nil segment Aug 7 21:39:36.405776361 (12h0m10.920878951s) [upload=12426]: Waiting for error group managing segments... Aug 7 21:39:36.423798896 (12h0m10.938901465s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.427938435 (12h0m10.943041015s) [noise=true , address=45.91.116.10:4092 , piece_id=GN66X2MISCTHDBGGDWOMPAD44VZHKZRFOHNGYP3CEVKHTQMEVS5Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.438590001 (12h0m10.953692572s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.439652545 (12h0m10.954755226s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.445868473 (12h0m10.960971037s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.446471740 (12h0m10.961574309s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.447690219 (12h0m10.96279279s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.449661572 (12h0m10.964764152s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.450416189 (12h0m10.965518772s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.453686307 (12h0m10.968788884s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.466878370 (12h0m10.981980947s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.543317192 (12h0m11.058419762s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.551730838 (12h0m11.066833415s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606002356 (12h0m11.121104933s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606032771 (12h0m11.121135347s) [seg_pos={0 0}, upload=12426]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:36.606628860 (12h0m11.121731425s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606708385 (12h0m11.121810956s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606769362 (12h0m11.121871933s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bgD3PwFrAz1UmQvd8V2rNAf7dFMbjrUEvn1ATTpSM85HijKHS): protocol: read tcp 159.223.143.39:49096->104.36.82.43:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:36.606779304 (12h0m11.121881873s) [noise=true , address=104.36.82.8:4004 , piece_id=DSETT3WILT5UL3OR6W27WV24QD2MOIYG4IN5CMNCQ2QJE26AMLAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection Aug 7 21:39:36.606819004 (12h0m11.121921573s) [seg_pos={0 0}, upload=12426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:36.606946009 (12h0m11.122048586s) [upload=12426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:36.606959527 (12h0m11.122062104s) [upload=12426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:08.632421584 (12h1m43.147524172s) [upload=12426]: Done uploading segments... Aug 7 21:41:08.632450790 (12h1m43.147553358s) [upload=12426]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:08.750931310 (12h1m43.266033893s) [upload=12453]: Starting upload Aug 7 21:41:08.751044034 (12h1m43.266146621s) [upload=12453]: Done starting upload Aug 7 21:41:08.751168536 (12h1m43.2662711s) [upload=12453]: Uploading segments... Aug 7 21:41:08.751366735 (12h1m43.266469311s) [upload=12453]: Got next segment. Inline: false Aug 7 21:41:08.751423163 (12h1m43.266525746s) [upload=12453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:08.751431978 (12h1m43.26653456s) [upload=12453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:08.718711666 (12h2m43.233814253s) [upload=12453]: Next segment err: context deadline exceeded Aug 7 21:42:08.718867335 (12h2m43.233969903s) [upload=12453]: Done uploading segments... Aug 7 21:42:08.718888660 (12h2m43.233991231s) [upload=12453]: Upload finished. err: context deadline exceeded Duration: 1.441120484s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.........XX...............] duration: 138.725339ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:45:15 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 138.755568ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXX.XX.X.] duration: 1.843887877s FAIL (2025-08-07 21:45:14 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.843943898s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...] duration: 751.870984ms FAIL (2025-08-07 21:45:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 751.891182ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXX.XX.X] duration: 30.001097843s FAIL (2025-08-07 21:44:57 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00112034s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 36 out of 60 is failed (threshold=3) [...................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...] duration: 256.458272ms FAIL (2025-08-07 21:44:56 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Duration: 256.483792ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 39 out of 60 is failed (threshold=3) [................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 1.044562411s FAIL (2025-08-07 21:44:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.044588381s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [...............................XXX........XX..XX............] duration: 97.192147ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:44:41 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 97.221152ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..] duration: 648.191249ms FAIL (2025-08-07 21:44:38 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 648.242628ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X.] duration: 427.108693ms FAIL (2025-08-07 21:44:35 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 427.162904ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.X] duration: 30.000654244s FAIL (2025-08-07 21:44:34 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000674836s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 300.291589ms FAIL (2025-08-07 21:44:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 300.319226ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.........XX..............] duration: 45.516126ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:44:28 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 45.536193ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..] duration: 745.30893ms FAIL (2025-08-07 21:44:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 745.394423ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [....................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X] duration: 30.001147957s FAIL (2025-08-07 21:44:22 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== tem: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:38.861083239 (12h3m13.37618582s) [upload=12477]: Done uploading segments... Aug 7 21:42:38.861105331 (12h3m13.3762081s) [upload=12477]: Upload finished. err: <nil> Aug 7 21:43:15.256120170 (12h3m49.771222746s) [upload=12489]: Starting upload Aug 7 21:43:15.256171635 (12h3m49.771274221s) [upload=12489]: Done starting upload Aug 7 21:43:15.256860901 (12h3m49.771963469s) [upload=12489]: Uploading segments... Aug 7 21:43:15.256905614 (12h3m49.7720082s) [upload=12489]: Got next segment. Inline: false Aug 7 21:43:15.257091199 (12h3m49.772193776s) [upload=12489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:15.257096064 (12h3m49.77219864s) [upload=12489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:43:15.326531147 (12h3m49.841633732s) [seg_pos={0 0}, upload=12489]: Begin upload segment... Aug 7 21:43:15.326629953 (12h3m49.841732541s) [noise=true , address=45.91.116.7:4009 , piece_id=WDX5H224KBK4FP722REIE3J46Z5FWYYPCVYSTVGTLM7S3FL25GIQ, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.330331305 (12h3m49.8454339s) [seg_pos={0 0}, upload=12489]: Done begin upload segment. Aug 7 21:43:15.330403853 (12h3m49.845506448s) [noise=true , address=45.91.116.13:4153 , piece_id=2UIYNBHPWUVU3LSWDJE437PT4MVXYAZBTZXAJGPMNSAP5PYYCL4Q, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.330931296 (12h3m49.846033894s) [noise=true , address=45.91.116.12:4225 , piece_id=LDIGJ3C3YMU5D62IMVZH2KW4ZQRZYK457EZ25LADRDXYPROMFF4A, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.331360287 (12h3m49.846462874s) [noise=true , address=45.91.116.10:4082 , piece_id=ZIHDRTMLZ6GGSTMFXUI3YWNFM2RGHCNI7QENDEUCHLAZWDBN7LOQ, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.331788211 (12h3m49.8468908s) [noise=true , address=104.36.82.6:4875 , piece_id=RND4VO45LAVLCIF6YWT6NKBHXEOXZGS45A36YWZ6QY7X66EHJYYA, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.332343288 (12h3m49.847445874s) [noise=true , address=77.90.31.15:4698 , piece_id=KJHVTSSAKIXBAYF3TTANMXCP5ZVVYRU2M7RT4YFQKFAWTRXS4RSA, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.332758696 (12h3m49.847861288s) [noise=true , address=45.91.116.14:4191 , piece_id=OSKV3JBJP4YGHWRPPC5UJLSLWZ6VRB34BWC3KCOZUF7UP2P42EAA, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.333363030 (12h3m49.848465612s) [noise=true , address=45.91.116.7:4011 , piece_id=BIVKBIHFQ75I5IFPGCL7FLBVQD2TMJE54PYGTLL62ZF7YOH3ZNNQ, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.334193088 (12h3m49.849295673s) [noise=true , address=77.90.31.10:4519 , piece_id=QKKT4CIYFOCLCJLFQE5BVAD7KQMHDCGQW6BA4D5IF2CSIO2LNFMQ, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.334840849 (12h3m49.849943436s) [noise=true , address=77.90.31.11:4554 , piece_id=XE7A2ZSGPT5RDO2HGGVSKE7BS22SAXLUEVKQB2JAEFIX2LMH4GLQ, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.335618818 (12h3m49.850721407s) [noise=true , address=45.91.116.5:4298 , piece_id=CZURDYWN5VNONLA4UYSYMLENOBKHEVJDLM5MAR6FBQCKOV6LWBTQ, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.336257067 (12h3m49.851359654s) [noise=true , address=45.91.116.4:4938 , piece_id=63PYYHJHUBH3OD2VQN6UQOWXEIY5A4J2BI4V5SOHJCNHNSUFORZA, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.336793163 (12h3m49.851895741s) [noise=true , address=45.91.116.6:4973 , piece_id=ZQVIYHIJASYLFZIKJPEMUGY7DK5ARWQTOIABYEBKRKNSAHE5CXHA, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.338370062 (12h3m49.853472653s) [noise=true , address=77.90.31.16:4735 , piece_id=ZITP52KZKEW7QW27USM42FLM6QQSM3EKOJBPCDEWC5DZ6MB7MTFA, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.340838071 (12h3m49.855940657s) [noise=true , address=77.90.31.21:4879 , piece_id=GG7A36OLLPBJBZYT6WHIBU5DIKBZ32C4Q5V6Z27VJIJMNWYNP5FA, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.342235865 (12h3m49.857338453s) [noise=true , address=45.91.116.8:4047 , piece_id=YSDFPJQUPVNED5FRDOXU4AOVR5DDEM64WB5GKYVENEUV57VMGCAA, seg_pos={0 0}, upload=12489]: Uploading piece... Aug 7 21:43:15.521168459 (12h3m50.036271022s) [upload=12489]: Next returned nil segment Aug 7 21:43:15.521187733 (12h3m50.036290296s) [upload=12489]: Waiting for error group managing segments... Aug 7 21:43:15.529103759 (12h3m50.044206352s) [noise=true , address=45.91.116.4:4938 , piece_id=63PYYHJHUBH3OD2VQN6UQOWXEIY5A4J2BI4V5SOHJCNHNSUFORZA, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.529164009 (12h3m50.044266585s) [noise=true , address=77.90.31.21:4879 , piece_id=GG7A36OLLPBJBZYT6WHIBU5DIKBZ32C4Q5V6Z27VJIJMNWYNP5FA, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.529622724 (12h3m50.044725295s) [noise=true , address=45.91.116.10:4082 , piece_id=ZIHDRTMLZ6GGSTMFXUI3YWNFM2RGHCNI7QENDEUCHLAZWDBN7LOQ, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.529630708 (12h3m50.044733304s) [noise=true , address=45.91.116.12:4225 , piece_id=LDIGJ3C3YMU5D62IMVZH2KW4ZQRZYK457EZ25LADRDXYPROMFF4A, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.530166156 (12h3m50.045268731s) [noise=true , address=45.91.116.13:4153 , piece_id=2UIYNBHPWUVU3LSWDJE437PT4MVXYAZBTZXAJGPMNSAP5PYYCL4Q, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.530311237 (12h3m50.045413816s) [noise=true , address=45.91.116.8:4047 , piece_id=YSDFPJQUPVNED5FRDOXU4AOVR5DDEM64WB5GKYVENEUV57VMGCAA, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.530714146 (12h3m50.045816722s) [noise=true , address=45.91.116.6:4973 , piece_id=ZQVIYHIJASYLFZIKJPEMUGY7DK5ARWQTOIABYEBKRKNSAHE5CXHA, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.531952022 (12h3m50.047054627s) [noise=true , address=77.90.31.11:4554 , piece_id=XE7A2ZSGPT5RDO2HGGVSKE7BS22SAXLUEVKQB2JAEFIX2LMH4GLQ, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.532166473 (12h3m50.047269042s) [noise=true , address=77.90.31.10:4519 , piece_id=QKKT4CIYFOCLCJLFQE5BVAD7KQMHDCGQW6BA4D5IF2CSIO2LNFMQ, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.533248610 (12h3m50.048351189s) [noise=true , address=45.91.116.7:4009 , piece_id=WDX5H224KBK4FP722REIE3J46Z5FWYYPCVYSTVGTLM7S3FL25GIQ, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.533780152 (12h3m50.048882731s) [noise=true , address=77.90.31.15:4698 , piece_id=KJHVTSSAKIXBAYF3TTANMXCP5ZVVYRU2M7RT4YFQKFAWTRXS4RSA, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.539352329 (12h3m50.054454921s) [noise=true , address=77.90.31.16:4735 , piece_id=ZITP52KZKEW7QW27USM42FLM6QQSM3EKOJBPCDEWC5DZ6MB7MTFA, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.540087872 (12h3m50.055190456s) [noise=true , address=45.91.116.7:4011 , piece_id=BIVKBIHFQ75I5IFPGCL7FLBVQD2TMJE54PYGTLL62ZF7YOH3ZNNQ, seg_pos={0 0}, upload=12489]: Done uploading piece. err: <nil> Aug 7 21:43:15.540103431 (12h3m50.055206011s) [seg_pos={0 0}, upload=12489]: Segment reached optimal threshold of 13 pieces. Aug 7 21:43:15.540229043 (12h3m50.055331625s) [noise=true , address=45.91.116.5:4298 , piece_id=CZURDYWN5VNONLA4UYSYMLENOBKHEVJDLM5MAR6FBQCKOV6LWBTQ, seg_pos={0 0}, upload=12489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:15.540346014 (12h3m50.055448594s) [noise=true , address=45.91.116.14:4191 , piece_id=OSKV3JBJP4YGHWRPPC5UJLSLWZ6VRB34BWC3KCOZUF7UP2P42EAA, seg_pos={0 0}, upload=12489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:15.540424151 (12h3m50.055526729s) [noise=true , address=104.36.82.6:4875 , piece_id=RND4VO45LAVLCIF6YWT6NKBHXEOXZGS45A36YWZ6QY7X66EHJYYA, seg_pos={0 0}, upload=12489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:15.540437431 (12h3m50.055540011s) [seg_pos={0 0}, upload=12489]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:43:15.540514709 (12h3m50.05561729s) [upload=12489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:43:15.540519651 (12h3m50.05562223s) [upload=12489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:15.607167115 (12h3m50.122269695s) [upload=12489]: Done uploading segments... Aug 7 21:43:15.607187034 (12h3m50.122289611s) [upload=12489]: Upload finished. err: <nil> Aug 7 21:43:52.199506676 (12h4m26.714609255s) [upload=12497]: Starting upload Aug 7 21:43:52.199573661 (12h4m26.71467623s) [upload=12497]: Done starting upload Aug 7 21:43:52.199632684 (12h4m26.71473526s) [upload=12497]: Uploading segments... Aug 7 21:43:52.199844109 (12h4m26.714946692s) [upload=12497]: Got next segment. Inline: false Aug 7 21:43:52.199899367 (12h4m26.715001952s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:52.199906793 (12h4m26.715009476s) [upload=12497]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:22.166742883 (12h4m56.681845464s) [upload=12497]: Next segment err: context deadline exceeded Aug 7 21:44:22.166864816 (12h4m56.681967396s) [upload=12497]: Done uploading segments... Aug 7 21:44:22.166885734 (12h4m56.681988313s) [upload=12497]: Upload finished. err: context deadline exceeded Duration:* 30.001190326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXX....] duration: 1.063734193s FAIL (2025-08-07 21:44:21 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g piece... Aug 7 21:39:36.313048033 (12h0m10.828150614s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313476014 (12h0m10.828578594s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313843993 (12h0m10.828946576s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314216946 (12h0m10.829319627s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314621829 (12h0m10.829724415s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314962361 (12h0m10.830064945s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315341892 (12h0m10.830444458s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315717142 (12h0m10.830819709s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316034230 (12h0m10.83113681s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316387250 (12h0m10.831489818s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316800121 (12h0m10.831902698s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317091487 (12h0m10.832194057s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317378915 (12h0m10.832481482s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317693529 (12h0m10.832796116s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318011241 (12h0m10.833113823s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318314555 (12h0m10.833417131s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.405758856 (12h0m10.920861443s) [upload=12426]: Next returned nil segment Aug 7 21:39:36.405776361 (12h0m10.920878951s) [upload=12426]: Waiting for error group managing segments... Aug 7 21:39:36.423798896 (12h0m10.938901465s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.427938435 (12h0m10.943041015s) [noise=true , address=45.91.116.10:4092 , piece_id=GN66X2MISCTHDBGGDWOMPAD44VZHKZRFOHNGYP3CEVKHTQMEVS5Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.438590001 (12h0m10.953692572s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.439652545 (12h0m10.954755226s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.445868473 (12h0m10.960971037s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.446471740 (12h0m10.961574309s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.447690219 (12h0m10.96279279s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.449661572 (12h0m10.964764152s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.450416189 (12h0m10.965518772s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.453686307 (12h0m10.968788884s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.466878370 (12h0m10.981980947s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.543317192 (12h0m11.058419762s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.551730838 (12h0m11.066833415s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606002356 (12h0m11.121104933s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606032771 (12h0m11.121135347s) [seg_pos={0 0}, upload=12426]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:36.606628860 (12h0m11.121731425s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606708385 (12h0m11.121810956s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606769362 (12h0m11.121871933s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bgD3PwFrAz1UmQvd8V2rNAf7dFMbjrUEvn1ATTpSM85HijKHS): protocol: read tcp 159.223.143.39:49096->104.36.82.43:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:36.606779304 (12h0m11.121881873s) [noise=true , address=104.36.82.8:4004 , piece_id=DSETT3WILT5UL3OR6W27WV24QD2MOIYG4IN5CMNCQ2QJE26AMLAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection Aug 7 21:39:36.606819004 (12h0m11.121921573s) [seg_pos={0 0}, upload=12426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:36.606946009 (12h0m11.122048586s) [upload=12426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:36.606959527 (12h0m11.122062104s) [upload=12426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:08.632421584 (12h1m43.147524172s) [upload=12426]: Done uploading segments... Aug 7 21:41:08.632450790 (12h1m43.147553358s) [upload=12426]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:08.750931310 (12h1m43.266033893s) [upload=12453]: Starting upload Aug 7 21:41:08.751044034 (12h1m43.266146621s) [upload=12453]: Done starting upload Aug 7 21:41:08.751168536 (12h1m43.2662711s) [upload=12453]: Uploading segments... Aug 7 21:41:08.751366735 (12h1m43.266469311s) [upload=12453]: Got next segment. Inline: false Aug 7 21:41:08.751423163 (12h1m43.266525746s) [upload=12453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:08.751431978 (12h1m43.26653456s) [upload=12453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:08.718711666 (12h2m43.233814253s) [upload=12453]: Next segment err: context deadline exceeded Aug 7 21:42:08.718867335 (12h2m43.233969903s) [upload=12453]: Done uploading segments... Aug 7 21:42:08.718888660 (12h2m43.233991231s) [upload=12453]: Upload finished. err: context deadline exceeded Duration: 1.06381241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.] duration: 5.771166896s FAIL (2025-08-07 21:44:21 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.771184505s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 1m0.001977098s FAIL (2025-08-07 21:44:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002005527s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [................................XXX........XX..XX...........] duration: 93.986658ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:44:14 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 94.047792ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 953.167316ms FAIL (2025-08-07 21:44:10 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 953.201138ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.........XX.............] duration: 62.591329ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:44:08 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 62.626501ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 36 out of 60 is failed (threshold=3) [....................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..] duration: 262.54548ms FAIL (2025-08-07 21:44:03 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Duration: 262.574269ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 29 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 240.450198ms FAIL (2025-08-07 21:44:02 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:51.494104017 (11h56m26.009206585s) [upload=12372]: Starting upload Aug 7 21:35:51.494158922 (11h56m26.009261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:43:02.575383061 (12h3m37.090485643s) [upload=12485]: Starting upload Aug 7 21:43:02.575486267 (12h3m37.090588853s) [upload=12485]: Done starting upload Aug 7 21:43:02.575568231 (12h3m37.090670811s) [upload=12485]: Uploading segments... Aug 7 21:43:02.575660495 (12h3m37.090763081s) [upload=12485]: Got next segment. Inline: true Aug 7 21:43:02.575667565 (12h3m37.090770151s) [upload=12485]: Waiting for error group managing segments... Aug 7 21:43:02.575741002 (12h3m37.090843584s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:02.575746898 (12h3m37.090849479s) [upload=12485]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:02.575752978 (12h3m37.090855558s) [upload=12485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:02.543664452 (12h4m37.058767018s) [upload=12485]: Done uploading segments... Aug 7 21:44:02.543687630 (12h4m37.058790215s) [upload=12485]: Upload finished. err: metaclient: context deadline exceeded Duration: 240.493824ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 548.209393ms FAIL (2025-08-07 21:44:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 548.237941ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 39 out of 60 is failed (threshold=3) [.................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 284.646844ms FAIL (2025-08-07 21:43:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 284.669641ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 36 out of 60 is failed (threshold=3) [.....................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.] duration: 220.615835ms FAIL (2025-08-07 21:43:38 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Duration: 220.654848ms core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.................................XXX........XX..XX..........] duration: 53.468605ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:43:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 53.490955ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.........XX............] duration: 41.784392ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:43:31 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 41.808739ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXX...] duration: 1.380956571s FAIL (2025-08-07 21:43:26 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g piece... Aug 7 21:39:36.313048033 (12h0m10.828150614s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313476014 (12h0m10.828578594s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313843993 (12h0m10.828946576s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314216946 (12h0m10.829319627s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314621829 (12h0m10.829724415s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314962361 (12h0m10.830064945s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315341892 (12h0m10.830444458s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315717142 (12h0m10.830819709s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316034230 (12h0m10.83113681s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316387250 (12h0m10.831489818s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316800121 (12h0m10.831902698s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317091487 (12h0m10.832194057s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317378915 (12h0m10.832481482s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317693529 (12h0m10.832796116s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318011241 (12h0m10.833113823s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318314555 (12h0m10.833417131s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.405758856 (12h0m10.920861443s) [upload=12426]: Next returned nil segment Aug 7 21:39:36.405776361 (12h0m10.920878951s) [upload=12426]: Waiting for error group managing segments... Aug 7 21:39:36.423798896 (12h0m10.938901465s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.427938435 (12h0m10.943041015s) [noise=true , address=45.91.116.10:4092 , piece_id=GN66X2MISCTHDBGGDWOMPAD44VZHKZRFOHNGYP3CEVKHTQMEVS5Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.438590001 (12h0m10.953692572s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.439652545 (12h0m10.954755226s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.445868473 (12h0m10.960971037s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.446471740 (12h0m10.961574309s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.447690219 (12h0m10.96279279s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.449661572 (12h0m10.964764152s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.450416189 (12h0m10.965518772s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.453686307 (12h0m10.968788884s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.466878370 (12h0m10.981980947s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.543317192 (12h0m11.058419762s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.551730838 (12h0m11.066833415s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606002356 (12h0m11.121104933s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606032771 (12h0m11.121135347s) [seg_pos={0 0}, upload=12426]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:36.606628860 (12h0m11.121731425s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606708385 (12h0m11.121810956s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606769362 (12h0m11.121871933s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bgD3PwFrAz1UmQvd8V2rNAf7dFMbjrUEvn1ATTpSM85HijKHS): protocol: read tcp 159.223.143.39:49096->104.36.82.43:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:36.606779304 (12h0m11.121881873s) [noise=true , address=104.36.82.8:4004 , piece_id=DSETT3WILT5UL3OR6W27WV24QD2MOIYG4IN5CMNCQ2QJE26AMLAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection Aug 7 21:39:36.606819004 (12h0m11.121921573s) [seg_pos={0 0}, upload=12426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:36.606946009 (12h0m11.122048586s) [upload=12426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:36.606959527 (12h0m11.122062104s) [upload=12426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:08.632421584 (12h1m43.147524172s) [upload=12426]: Done uploading segments... Aug 7 21:41:08.632450790 (12h1m43.147553358s) [upload=12426]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:08.750931310 (12h1m43.266033893s) [upload=12453]: Starting upload Aug 7 21:41:08.751044034 (12h1m43.266146621s) [upload=12453]: Done starting upload Aug 7 21:41:08.751168536 (12h1m43.2662711s) [upload=12453]: Uploading segments... Aug 7 21:41:08.751366735 (12h1m43.266469311s) [upload=12453]: Got next segment. Inline: false Aug 7 21:41:08.751423163 (12h1m43.266525746s) [upload=12453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:08.751431978 (12h1m43.26653456s) [upload=12453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:08.718711666 (12h2m43.233814253s) [upload=12453]: Next segment err: context deadline exceeded Aug 7 21:42:08.718867335 (12h2m43.233969903s) [upload=12453]: Done uploading segments... Aug 7 21:42:08.718888660 (12h2m43.233991231s) [upload=12453]: Upload finished. err: context deadline exceeded Duration: 1.380984658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXX.XX.] duration: 2.058489068s FAIL (2025-08-07 21:43:25 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.058523864s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXX.XX] duration: 30.400965181s FAIL (2025-08-07 21:43:23 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.400996032s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.001246341s FAIL (2025-08-07 21:43:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00126782s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 39 out of 60 is failed (threshold=3) [..................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 12.491198143s FAIL (2025-08-07 21:43:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.491213857s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.00318853s FAIL (2025-08-07 21:43:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests Duration: 1.00321096s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.....................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 574.629503ms FAIL (2025-08-07 21:43:15 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 12h1m14.02027627s) [upload=12435]: Upload finished. err: context deadline exceeded Aug 7 21:41:00.056743511 (12h1m34.571846093s) [upload=12451]: Starting upload Aug 7 21:41:00.056834896 (12h1m34.571937476s) [upload=12451]: Done starting upload Aug 7 21:41:00.056947293 (12h1m34.572049861s) [upload=12451]: Uploading segments... Aug 7 21:41:00.057253002 (12h1m34.572355591s) [upload=12451]: Got next segment. Inline: false Aug 7 21:41:00.057314022 (12h1m34.572416588s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:00.057326500 (12h1m34.572429079s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:06.802724338 (12h1m41.317826923s) [seg_pos={0 0}, upload=12451]: Begin upload segment... Aug 7 21:41:06.802779366 (12h1m41.317881933s) [seg_pos={0 0}, upload=12451]: Done begin upload segment. Aug 7 21:41:06.802802866 (12h1m41.317905445s) [noise=true , address=77.90.31.14:4663 , piece_id=MCP22S4S2267OULHW5SYQ6O7I3LG4PKWIMEQ45MS2KA4JOAHFY6A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.802894905 (12h1m41.317997502s) [noise=true , address=77.90.31.13:4627 , piece_id=FHLWRN7K4OU6HDQQ6X44TMLI7GPQPDJ4INQGGKMMCYSCSYGPR33Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803391882 (12h1m41.318494468s) [noise=true , address=104.36.82.10:4078 , piece_id=7JWKJPKOUISJF2EXKBWXMPCULDIUJRKZUVA72FOLY32PHJFVRD2Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803522360 (12h1m41.31862495s) [noise=true , address=104.36.82.15:4258 , piece_id=NF3KM2RU3RLV45BOPW46NW3UGL32WH473ADDXFROB2IQC3I54XGA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.803904747 (12h1m41.31900733s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSU62T7KZCWTJ64CSFXAOMTFB3DY7QU67JWANKCHA64EYGJPU2GQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804010771 (12h1m41.319113359s) [noise=true , address=77.90.31.11:4553 , piece_id=Y56CRBHKAGCWU2PEKQIMAMGZ6TNMAWCSHIMCOQUEK7MK64EVWE4A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804197608 (12h1m41.319300187s) [noise=true , address=104.36.82.6:4875 , piece_id=5ZGFKU7IVKNZWIIW3A2E6S6ZVQMDHOYDUKGRWFL3YEMK3ZI47DEQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804479589 (12h1m41.319582176s) [noise=true , address=104.36.82.13:4186 , piece_id=P7Y3FZQF2T5Y6CRLKY22OOYZPJRUECW7VEYNCPMZCCWPH2OM47OQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804499574 (12h1m41.31960214s) [noise=true , address=77.90.31.21:4878 , piece_id=LVQCIDDIOFQA2RKX7HS5SRUBOWO7IIMD7IVQF7Y5OI3T3LDUGL2A, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.804953768 (12h1m41.320056351s) [noise=true , address=45.91.116.7:4009 , piece_id=OZXLMKK7LWVAAPIUI6WZ6WLYKDNKXA4T5PJG34JNUEU3A7ALAG3Q, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805026844 (12h1m41.320129434s) [noise=true , address=104.36.82.3:4767 , piece_id=GB7R5UALDIVEKCEFGWJMY4JJUPPNGSGUUJIQZYBHUKXRY4DCG3VA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805236078 (12h1m41.320338659s) [noise=true , address=45.91.116.6:4974 , piece_id=CSR4RAGCHGZ3AHYDZLQT4OFW7TLI5IEJHWD5Z6IGRG5ZKQGU3UFA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805491522 (12h1m41.320594111s) [noise=true , address=45.91.116.12:4227 , piece_id=5J6UANJSVQRV3A3HMPM6FETPYHZKFV3OLL7DLCV6WY3WENLTTGTA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805532740 (12h1m41.320635319s) [noise=true , address=45.91.116.13:4155 , piece_id=SVJJFQA5ZMIMHVT7E5JTF2S7PONZWD6S6T7WUYS63UH6MTOFNDZQ, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.805966784 (12h1m41.321069381s) [noise=true , address=104.36.82.9:4005 , piece_id=HTJDYQJBCBGMVUWCEHH6CW5PVXRUAYAEYU42QYL4ZINXLWP66KQA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.806546714 (12h1m41.321649292s) [noise=true , address=45.91.116.5:4298 , piece_id=JWYNJUQ3MQFCC5W4L7QBXU2JR6OJQH5GPML6SMNUZWNO2LLYK7LA, seg_pos={0 0}, upload=12451]: Uploading piece... Aug 7 21:41:06.926689944 (12h1m41.441792523s) [upload=12451]: Next returned nil segment Aug 7 21:41:06.926709145 (12h1m41.441811733s) [upload=12451]: Waiting for error group managing segments... Aug 7 21:41:06.931954878 (12h1m41.447057458s) [noise=true , address=45.91.116.6:4974 , piece_id=CSR4RAGCHGZ3AHYDZLQT4OFW7TLI5IEJHWD5Z6IGRG5ZKQGU3UFA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.933221632 (12h1m41.448324213s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSU62T7KZCWTJ64CSFXAOMTFB3DY7QU67JWANKCHA64EYGJPU2GQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.933245521 (12h1m41.448348097s) [noise=true , address=45.91.116.12:4227 , piece_id=5J6UANJSVQRV3A3HMPM6FETPYHZKFV3OLL7DLCV6WY3WENLTTGTA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.935433763 (12h1m41.450536353s) [noise=true , address=77.90.31.14:4663 , piece_id=MCP22S4S2267OULHW5SYQ6O7I3LG4PKWIMEQ45MS2KA4JOAHFY6A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.935767291 (12h1m41.450869868s) [noise=true , address=45.91.116.13:4155 , piece_id=SVJJFQA5ZMIMHVT7E5JTF2S7PONZWD6S6T7WUYS63UH6MTOFNDZQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.938400107 (12h1m41.453502685s) [noise=true , address=104.36.82.13:4186 , piece_id=P7Y3FZQF2T5Y6CRLKY22OOYZPJRUECW7VEYNCPMZCCWPH2OM47OQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.938687716 (12h1m41.453790278s) [noise=true , address=45.91.116.5:4298 , piece_id=JWYNJUQ3MQFCC5W4L7QBXU2JR6OJQH5GPML6SMNUZWNO2LLYK7LA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.939447304 (12h1m41.454549875s) [noise=true , address=77.90.31.21:4878 , piece_id=LVQCIDDIOFQA2RKX7HS5SRUBOWO7IIMD7IVQF7Y5OI3T3LDUGL2A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.939805604 (12h1m41.454908182s) [noise=true , address=104.36.82.6:4875 , piece_id=5ZGFKU7IVKNZWIIW3A2E6S6ZVQMDHOYDUKGRWFL3YEMK3ZI47DEQ, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.950100367 (12h1m41.465202944s) [noise=true , address=45.91.116.7:4009 , piece_id=OZXLMKK7LWVAAPIUI6WZ6WLYKDNKXA4T5PJG34JNUEU3A7ALAG3Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.950539445 (12h1m41.465642035s) [noise=true , address=77.90.31.13:4627 , piece_id=FHLWRN7K4OU6HDQQ6X44TMLI7GPQPDJ4INQGGKMMCYSCSYGPR33Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953496508 (12h1m41.468599086s) [noise=true , address=77.90.31.11:4553 , piece_id=Y56CRBHKAGCWU2PEKQIMAMGZ6TNMAWCSHIMCOQUEK7MK64EVWE4A, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953532645 (12h1m41.468635227s) [noise=true , address=104.36.82.10:4078 , piece_id=7JWKJPKOUISJF2EXKBWXMPCULDIUJRKZUVA72FOLY32PHJFVRD2Q, seg_pos={0 0}, upload=12451]: Done uploading piece. err: <nil> Aug 7 21:41:06.953548888 (12h1m41.468651465s) [seg_pos={0 0}, upload=12451]: Segment reached optimal threshold of 13 pieces. Aug 7 21:41:06.953680391 (12h1m41.468782969s) [noise=true , address=104.36.82.15:4258 , piece_id=NF3KM2RU3RLV45BOPW46NW3UGL32WH473ADDXFROB2IQC3I54XGA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:06.953699436 (12h1m41.468802002s) [noise=true , address=104.36.82.3:4767 , piece_id=GB7R5UALDIVEKCEFGWJMY4JJUPPNGSGUUJIQZYBHUKXRY4DCG3VA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:06.953979656 (12h1m41.46908222s) [noise=true , address=104.36.82.9:4005 , piece_id=HTJDYQJBCBGMVUWCEHH6CW5PVXRUAYAEYU42QYL4ZINXLWP66KQA, seg_pos={0 0}, upload=12451]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: read tcp 159.223.143.39:53626->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:41:06.953998020 (12h1m41.469100586s) [seg_pos={0 0}, upload=12451]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:41:06.954079225 (12h1m41.469181809s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:06.954085123 (12h1m41.46918771s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:31.504450541 (12h2m6.019553121s) [upload=12451]: Done uploading segments... Aug 7 21:41:31.504481231 (12h2m6.019583808s) [upload=12451]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:31.625936603 (12h2m6.141039168s) [upload=12459]: Starting upload Aug 7 21:41:31.626009741 (12h2m6.141112327s) [upload=12459]: Done starting upload Aug 7 21:41:31.626253438 (12h2m6.141356016s) [upload=12459]: Uploading segments... Aug 7 21:41:31.626287931 (12h2m6.141390513s) [upload=12459]: Got next segment. Inline: false Aug 7 21:41:31.626319906 (12h2m6.141422482s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:31.626323686 (12h2m6.141426262s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:01.589273243 (12h2m36.104375832s) [upload=12459]: Next segment err: context deadline exceeded Aug 7 21:42:01.589453960 (12h2m36.104556539s) [upload=12459]: Done uploading segments... Aug 7 21:42:01.589475808 (12h2m36.104578383s) [upload=12459]: Upload finished. err: context deadline exceeded Duration: 574.653274ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 36 out of 60 is failed (threshold=3) [......................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.001306025s FAIL (2025-08-07 21:43:13 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:26.814928602 (12h0m1.330031169s) [upload=12425]: Starting upload Aug 7 21:39:26.815013293 (12h0m1.330115869s) [upload=12425]: Done starting upload Aug 7 21:39:26.815107574 (12h0m1.330210142s) [upload=12425]: Uploading segments... Aug 7 21:39:26.815155995 (12h0m1.33025858s) [upload=12425]: Got next segment. Inline: true Aug 7 21:39:26.815160616 (12h0m1.330263185s) [upload=12425]: Waiting for error group managing segments... Aug 7 21:39:26.815211931 (12h0m1.330314498s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:26.815216944 (12h0m1.330319523s) [upload=12425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:26.815220179 (12h0m1.330322746s) [upload=12425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.783158340 (12h0m31.298260907s) [upload=12425]: Done uploading segments... Aug 7 21:39:56.783182464 (12h0m31.29828504s) [upload=12425]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:18.713184837 (12h0m53.22828742s) [upload=12438]: Starting upload Aug 7 21:40:18.713271194 (12h0m53.228373784s) [upload=12438]: Done starting upload Aug 7 21:40:18.713355385 (12h0m53.228457956s) [upload=12438]: Uploading segments... Aug 7 21:40:18.713472939 (12h0m53.228575524s) [upload=12438]: Got next segment. Inline: true Aug 7 21:40:18.713478436 (12h0m53.228581015s) [upload=12438]: Waiting for error group managing segments... Aug 7 21:40:18.713541897 (12h0m53.228644474s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:18.713546499 (12h0m53.228649078s) [upload=12438]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:18.713549918 (12h0m53.228652501s) [upload=12438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:48.497408400 (12h1m23.012510976s) [upload=12438]: Done uploading segments... Aug 7 21:40:48.497429106 (12h1m23.012531683s) [upload=12438]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:17.515953647 (12h1m52.031056228s) [upload=12456]: Starting upload Aug 7 21:41:17.516045246 (12h1m52.031147827s) [upload=12456]: Done starting upload Aug 7 21:41:17.516110459 (12h1m52.031213028s) [upload=12456]: Uploading segments... Aug 7 21:41:17.516209063 (12h1m52.031311648s) [upload=12456]: Got next segment. Inline: true Aug 7 21:41:17.516214550 (12h1m52.031317127s) [upload=12456]: Waiting for error group managing segments... Aug 7 21:41:17.516275557 (12h1m52.031378123s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:17.516280518 (12h1m52.031383089s) [upload=12456]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:17.516284057 (12h1m52.031386633s) [upload=12456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:34.121376239 (12h2m8.636478806s) [upload=12456]: Done uploading segments... Aug 7 21:41:34.121392703 (12h2m8.636495269s) [upload=12456]: Upload finished. err: <nil> Aug 7 21:41:44.147890574 (12h2m18.662993154s) [upload=12462]: Starting upload Aug 7 21:41:44.147968937 (12h2m18.663071514s) [upload=12462]: Done starting upload Aug 7 21:41:44.148048103 (12h2m18.663150671s) [upload=12462]: Uploading segments... Aug 7 21:41:44.148093740 (12h2m18.663196325s) [upload=12462]: Got next segment. Inline: true Aug 7 21:41:44.148116087 (12h2m18.663218664s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:41:44.148162872 (12h2m18.663265448s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:44.148167364 (12h2m18.66326994s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:44.148170643 (12h2m18.663273219s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:14.092757905 (12h2m48.607860482s) [upload=12462]: Done uploading segments... Aug 7 21:42:14.092790048 (12h2m48.607892623s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:43.144431091 (12h3m17.659533678s) [upload=12479]: Starting upload Aug 7 21:42:43.144520195 (12h3m17.659622762s) [upload=12479]: Done starting upload Aug 7 21:42:43.144685088 (12h3m17.659787664s) [upload=12479]: Uploading segments... Aug 7 21:42:43.144725439 (12h3m17.659828024s) [upload=12479]: Got next segment. Inline: true Aug 7 21:42:43.144730400 (12h3m17.659832976s) [upload=12479]: Waiting for error group managing segments... Aug 7 21:42:43.144781566 (12h3m17.659884142s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:43.144787556 (12h3m17.659890132s) [upload=12479]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:43.144790885 (12h3m17.659893464s) [upload=12479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:13.111481181 (12h3m47.626583759s) [upload=12479]: Done uploading segments... Aug 7 21:43:13.111503700 (12h3m47.62660628s) [upload=12479]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001326445s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 1m0.001911929s FAIL (2025-08-07 21:43:03 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00194963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001226249s FAIL (2025-08-07 21:43:02 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:51.494104017 (11h56m26.009206585s) [upload=12372]: Starting upload Aug 7 21:35:51.494158922 (11h56m26.009261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:51.724657874 (11h59m26.239760457s) [upload=12415]: Starting upload Aug 7 21:38:51.724746034 (11h59m26.239848614s) [upload=12415]: Done starting upload Aug 7 21:38:51.724840321 (11h59m26.239942884s) [upload=12415]: Uploading segments... Aug 7 21:38:51.724890624 (11h59m26.239993212s) [upload=12415]: Got next segment. Inline: true Aug 7 21:38:51.724895839 (11h59m26.239998417s) [upload=12415]: Waiting for error group managing segments... Aug 7 21:38:51.724943298 (11h59m26.240045863s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:51.724950304 (11h59m26.240052883s) [upload=12415]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:51.724954010 (11h59m26.240056592s) [upload=12415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:51.695792970 (12h0m26.210895538s) [upload=12415]: Done uploading segments... Aug 7 21:39:51.695812586 (12h0m26.210915161s) [upload=12415]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.414009593 (12h0m30.929112171s) [upload=12430]: Starting upload Aug 7 21:39:56.414099284 (12h0m30.92920186s) [upload=12430]: Done starting upload Aug 7 21:39:56.414199233 (12h0m30.929301808s) [upload=12430]: Uploading segments... Aug 7 21:39:56.414265867 (12h0m30.929368444s) [upload=12430]: Got next segment. Inline: true Aug 7 21:39:56.414271777 (12h0m30.929374359s) [upload=12430]: Waiting for error group managing segments... Aug 7 21:39:56.414322440 (12h0m30.929425017s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.414327733 (12h0m30.92943031s) [upload=12430]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.414334126 (12h0m30.929436704s) [upload=12430]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:56.384589677 (12h1m30.899692241s) [upload=12430]: Done uploading segments... Aug 7 21:40:56.384608625 (12h1m30.899711188s) [upload=12430]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:02.421461554 (12h1m36.936564122s) [upload=12452]: Starting upload Aug 7 21:41:02.421531078 (12h1m36.936633656s) [upload=12452]: Done starting upload Aug 7 21:41:02.421590963 (12h1m36.936693533s) [upload=12452]: Uploading segments... Aug 7 21:41:02.421631066 (12h1m36.936733653s) [upload=12452]: Got next segment. Inline: true Aug 7 21:41:02.421635882 (12h1m36.936738528s) [upload=12452]: Waiting for error group managing segments... Aug 7 21:41:02.421687586 (12h1m36.93679017s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:02.421692815 (12h1m36.936795397s) [upload=12452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:02.421696268 (12h1m36.936798846s) [upload=12452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:02.390011657 (12h2m36.905114241s) [upload=12452]: Done uploading segments... Aug 7 21:42:02.390037739 (12h2m36.905140318s) [upload=12452]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:02.517855493 (12h2m37.032958077s) [upload=12466]: Starting upload Aug 7 21:42:02.517972469 (12h2m37.033075039s) [upload=12466]: Done starting upload Aug 7 21:42:02.518057838 (12h2m37.03316042s) [upload=12466]: Uploading segments... Aug 7 21:42:02.518119646 (12h2m37.033222293s) [upload=12466]: Got next segment. Inline: true Aug 7 21:42:02.518126847 (12h2m37.033229435s) [upload=12466]: Waiting for error group managing segments... Aug 7 21:42:02.518187158 (12h2m37.033289743s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:02.518194415 (12h2m37.033296998s) [upload=12466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:02.518199250 (12h2m37.033301835s) [upload=12466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.481873729 (12h3m36.996976297s) [upload=12466]: Done uploading segments... Aug 7 21:43:02.481894284 (12h3m36.996996861s) [upload=12466]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001242602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 35 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.] duration: 253.654957ms FAIL (2025-08-07 21:43:01 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 253.680353ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.015597733s FAIL (2025-08-07 21:43:00 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad Aug 7 21:41:10.897416927 (12h1m45.412519497s) [upload=12454]: Uploading segments... Aug 7 21:41:10.897474206 (12h1m45.412576797s) [upload=12454]: Got next segment. Inline: false Aug 7 21:41:10.897513736 (12h1m45.412616324s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:10.897519282 (12h1m45.412621859s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:11.500021598 (12h1m46.015124178s) [seg_pos={0 0}, upload=12454]: Begin upload segment... Aug 7 21:41:11.500079622 (12h1m46.015182188s) [seg_pos={0 0}, upload=12454]: Done begin upload segment. Aug 7 21:41:11.500118772 (12h1m46.015221351s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.500806701 (12h1m46.015909286s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501393097 (12h1m46.016495684s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.501864237 (12h1m46.016966823s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502397272 (12h1m46.017499856s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.502991250 (12h1m46.018093833s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503068626 (12h1m46.018171213s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.503812967 (12h1m46.01891555s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.504251191 (12h1m46.019353771s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.507497110 (12h1m46.022599695s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510027508 (12h1m46.025130095s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.510608814 (12h1m46.025711387s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511293314 (12h1m46.026395881s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.511955184 (12h1m46.027057752s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.512647119 (12h1m46.02774969s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.514540714 (12h1m46.02964329s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.516459673 (12h1m46.03156225s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.518440797 (12h1m46.033543374s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Uploading piece... Aug 7 21:41:11.601237057 (12h1m46.116339633s) [upload=12454]: Next returned nil segment Aug 7 21:41:11.601254163 (12h1m46.116356748s) [upload=12454]: Waiting for error group managing segments... Aug 7 21:41:11.610393629 (12h1m46.125496199s) [noise=true , address=45.91.116.14:4197 , piece_id=W6N7ZLPNF63UWKUPQXVUYNRIWMLHB3XFV7ZIRP3ORGX5PD3STYFA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.616609371 (12h1m46.131711941s) [noise=true , address=104.36.82.38:4699 , piece_id=LVUDND2EOCGHX2EPB4Y2HP3Q2NYCQCHJGYNJX5H5H66WRNVVZ4GA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.625318997 (12h1m46.140421565s) [noise=true , address=104.36.82.16:4327 , piece_id=6UF5YG7P3FXSOEGT33GJLKWDG6IYWV74KG6T55HVSMKRCPYSA6RQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626336516 (12h1m46.141439093s) [noise=true , address=104.36.82.61:4391 , piece_id=ZZZ7TMMANVQ43S35BH4YZ47IH6TYW7PX5PPC67LOLTMRHTXWQDNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.626937669 (12h1m46.142040238s) [noise=true , address=104.36.82.59:4314 , piece_id=DGMX6JW3T22DDILNCCRDOMDHFMY3Q4ANT7FTE4HZ67WCN2GJVHHA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.631425392 (12h1m46.146527958s) [noise=true , address=104.36.82.46:4503 , piece_id=ROX3KIN5BCDF23PATKXN2N2FSJYL7QU2M6ITYHVUPCUY7ZRDC3NA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.632916680 (12h1m46.148019247s) [noise=true , address=104.36.82.4:4821 , piece_id=TSLRPKUADUFL76ELOGMJE3MIXDFHI3PMUI2MAXA26HWBYF7YXTJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.634385568 (12h1m46.149488145s) [noise=true , address=104.36.82.44:4896 , piece_id=SMJTZX6NPKUAHUGHP6EQ2MBC4XOQGKU6EEAHKTVSVRJL7BOUQRNA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.642028486 (12h1m46.157131052s) [noise=true , address=104.36.82.22:4519 , piece_id=QB7RNV4UHFOS7N2HGPJBPIZZDSUAZRJELPQGHWYVMQ4OYAVM5KYA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.659779623 (12h1m46.174882199s) [noise=true , address=104.36.82.23:4469 , piece_id=EKGBG2JP2KX2DSDIRXEOBAC24ZTJMNXSCSUAFFTKZUK2F72G7NOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.799649296 (12h1m46.314751866s) [noise=true , address=79.127.201.215:4354 , piece_id=NHT7RBK4IUQIWAUFAHWJTTTMYYYYAH6BSK2SYDUVKIR73SL6AQOA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.860021788 (12h1m46.375124358s) [noise=true , address=104.36.82.14:4249 , piece_id=Y3VEW6MYUQ3DPEQ7VOW46ZAVD7WUIUTYNOJXDTZJYFULLRC2HR7A, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:11.988336713 (12h1m46.503439295s) [noise=true , address=104.36.82.15:4288 , piece_id=UPEQULL3FHKMG2TYXAUAA4KSNTFNIPADFABQEUENIUBO6S7T2KGQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071883376 (12h1m46.586985938s) [noise=true , address=79.127.205.245:4221 , piece_id=JUQLSSYHRXYHJFTPRTJNITVVAPJPBGMEBKFQ5UHDIM25YHKG6JKA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: <nil> Aug 7 21:41:12.071916823 (12h1m46.587019403s) [seg_pos={0 0}, upload=12454]: Segment reached optimal threshold of 14 pieces. Aug 7 21:41:12.072047118 (12h1m46.587149696s) [noise=true , address=64.49.13.169:4706 , piece_id=RU2A7ZDIBGIYOOWOSJQ4C3HKXKF7GHQKBKLUNF2RFLUK2H7IEFJA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072172149 (12h1m46.587274717s) [noise=true , address=64.49.13.170:4725 , piece_id=I6RVXAWKZ4RLK5U3DMIRRIL3TXAV6T2ORFH4XULBS2ZTMOU235SA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072214360 (12h1m46.587316943s) [noise=true , address=104.36.82.5:4841 , piece_id=AN7EELTOV2BUCNGBB3WNY2XZGFN2PVMLXXAI2RSWDN2ZQ62KE2OA, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:12.072250829 (12h1m46.587353397s) [noise=true , address=104.36.82.37:4669 , piece_id=CT55W7LIVJM3I2FKDVDEEZXUSQRLJLWQFJPRBCGJ465ISOOGJNWQ, seg_pos={0 0}, upload=12454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1UCF4dQobQEVqe9XB5tKxDx9L2P7kkQW179fxc1ir9ZtQVsM): protocol: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36108->104.36.82.37:4669: use of closed network connection Aug 7 21:41:12.072265693 (12h1m46.58736827s) [seg_pos={0 0}, upload=12454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:41:12.072353087 (12h1m46.58745567s) [upload=12454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:12.072358389 (12h1m46.587460973s) [upload=12454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:12.917439625 (12h1m47.432542188s) [upload=12454]: Done uploading segments... Aug 7 21:41:12.917479283 (12h1m47.43258186s) [upload=12454]: Upload finished. err: <nil> Duration: 1.015626784s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 39 out of 60 is failed (threshold=3) [...................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001444724s FAIL (2025-08-07 21:43:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001491362s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.00124219s FAIL (2025-08-07 21:42:47 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001257367s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXX..] duration: 16.155281241s FAIL (2025-08-07 21:42:44 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g piece... Aug 7 21:39:36.313048033 (12h0m10.828150614s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313476014 (12h0m10.828578594s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.313843993 (12h0m10.828946576s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314216946 (12h0m10.829319627s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314621829 (12h0m10.829724415s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.314962361 (12h0m10.830064945s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315341892 (12h0m10.830444458s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.315717142 (12h0m10.830819709s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316034230 (12h0m10.83113681s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316387250 (12h0m10.831489818s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.316800121 (12h0m10.831902698s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317091487 (12h0m10.832194057s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317378915 (12h0m10.832481482s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.317693529 (12h0m10.832796116s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318011241 (12h0m10.833113823s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.318314555 (12h0m10.833417131s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Uploading piece... Aug 7 21:39:36.405758856 (12h0m10.920861443s) [upload=12426]: Next returned nil segment Aug 7 21:39:36.405776361 (12h0m10.920878951s) [upload=12426]: Waiting for error group managing segments... Aug 7 21:39:36.423798896 (12h0m10.938901465s) [noise=true , address=104.36.82.62:4410 , piece_id=5FJYFWTT6URYIL3VOB4JHSSD6MTOPAVKZOGVGCUAGAOIIJ2UBGYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.427938435 (12h0m10.943041015s) [noise=true , address=45.91.116.10:4092 , piece_id=GN66X2MISCTHDBGGDWOMPAD44VZHKZRFOHNGYP3CEVKHTQMEVS5Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.438590001 (12h0m10.953692572s) [noise=true , address=104.36.82.45:4938 , piece_id=BZ6P3IRTXLDHZQAAZOKDDUH25ADXYNGJFIZHCVAI6I3E4PMMFRYA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.439652545 (12h0m10.954755226s) [noise=true , address=104.36.82.47:4520 , piece_id=I7DK7V7UFL5SIOAJLPZ4RMVU6B6JLNB6RUKYKZ3D24OD547N3VYQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.445868473 (12h0m10.960971037s) [noise=true , address=104.36.82.35:4604 , piece_id=QB4DEPZYMW4EZETFAZHTYSETDWIXEVVJRWZXYIKGXUAJ3DE6Z7EQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.446471740 (12h0m10.961574309s) [noise=true , address=104.36.82.44:4883 , piece_id=ZB7SYIYKQRLBAFKLRLS4GXLJUI6RMUIIK44Y2SU4EYMOMNDGBSBQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.447690219 (12h0m10.96279279s) [noise=true , address=104.36.82.46:4483 , piece_id=U2HLYO3HYHAKAZH2S54NH37L3VVRKEKYZXYXY36HXKGDSBBDA5SA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.449661572 (12h0m10.964764152s) [noise=true , address=104.36.82.55:4194 , piece_id=UPYBZNKYLA4BBYJLLPIPDA25YBYXDICSGK4WXOTDPXWMX6XRRDAQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.450416189 (12h0m10.965518772s) [noise=true , address=45.91.116.12:4259 , piece_id=ZKH4H35PBL3JCE4X2BNWMVWEGMOELZCAQ2OWEYTPQBZ6K6CVWAFA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.453686307 (12h0m10.968788884s) [noise=true , address=104.36.82.12:4165 , piece_id=LE2Q7VMSL5BZO5YEYUDZDF3FJZINWTQXKGRIWDCI2OMB25VGMJAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.466878370 (12h0m10.981980947s) [noise=true , address=104.36.82.39:4759 , piece_id=ILWISBDAOTYGAVXZUSBRGKKGEHOT74YEANQBM5HVVBTWU44OOZ4Q, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.543317192 (12h0m11.058419762s) [noise=true , address=104.36.82.58:4272 , piece_id=ULY4KMVWODOIMTMJOX65CBXICL3WLPP4S27SA2Q5MPQSEG4X3DGQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.551730838 (12h0m11.066833415s) [noise=true , address=104.36.82.17:4358 , piece_id=LDDM4AICXAQUNV44MO6R72S7GEWISAFRCJ3EGZNEWE3KTKRWGKVQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606002356 (12h0m11.121104933s) [noise=true , address=79.127.201.215:4359 , piece_id=VIYNP2JVDUNUYAEWT3Z5FGZK46CWEIUNVTFLEXEOTD3MZSKWTXFQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: <nil> Aug 7 21:39:36.606032771 (12h0m11.121135347s) [seg_pos={0 0}, upload=12426]: Segment reached optimal threshold of 14 pieces. Aug 7 21:39:36.606628860 (12h0m11.121731425s) [noise=true , address=104.36.82.46:4491 , piece_id=S3D225YPJTRF4EMDVEWSNMX2PC4XK34V6OWXIGOBK46PRZ2U5V4A, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606708385 (12h0m11.121810956s) [noise=true , address=104.36.82.60:4369 , piece_id=NG3HRY4AYC2Y3L763WTMUFO2YDH6VJGQSAOWULJJQ6ERHFEPZEKQ, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:36.606769362 (12h0m11.121871933s) [noise=true , address=104.36.82.43:4845 , piece_id=UU6NEQ4YTKKELZJ4NA4LLMMQRJ6ZAC77A5SMKIO6QOWFA7G7BGRA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bgD3PwFrAz1UmQvd8V2rNAf7dFMbjrUEvn1ATTpSM85HijKHS): protocol: read tcp 159.223.143.39:49096->104.36.82.43:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:39:36.606779304 (12h0m11.121881873s) [noise=true , address=104.36.82.8:4004 , piece_id=DSETT3WILT5UL3OR6W27WV24QD2MOIYG4IN5CMNCQ2QJE26AMLAA, seg_pos={0 0}, upload=12426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34956->104.36.82.8:4004: use of closed network connection Aug 7 21:39:36.606819004 (12h0m11.121921573s) [seg_pos={0 0}, upload=12426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:39:36.606946009 (12h0m11.122048586s) [upload=12426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:36.606959527 (12h0m11.122062104s) [upload=12426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:08.632421584 (12h1m43.147524172s) [upload=12426]: Done uploading segments... Aug 7 21:41:08.632450790 (12h1m43.147553358s) [upload=12426]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:08.750931310 (12h1m43.266033893s) [upload=12453]: Starting upload Aug 7 21:41:08.751044034 (12h1m43.266146621s) [upload=12453]: Done starting upload Aug 7 21:41:08.751168536 (12h1m43.2662711s) [upload=12453]: Uploading segments... Aug 7 21:41:08.751366735 (12h1m43.266469311s) [upload=12453]: Got next segment. Inline: false Aug 7 21:41:08.751423163 (12h1m43.266525746s) [upload=12453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:08.751431978 (12h1m43.26653456s) [upload=12453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:08.718711666 (12h2m43.233814253s) [upload=12453]: Next segment err: context deadline exceeded Aug 7 21:42:08.718867335 (12h2m43.233969903s) [upload=12453]: Done uploading segments... Aug 7 21:42:08.718888660 (12h2m43.233991231s) [upload=12453]: Upload finished. err: context deadline exceeded Duration: 16.155311645s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)