Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 14.85730947s FAIL (2025-06-22 23:44:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.857397642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X............................X.......X...................] duration: 13.477114478s FAIL (2025-06-22 23:44:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.477146019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 27.151515038s FAIL (2025-06-22 23:44:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.151554692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X............................X.......X..................] duration: 19.768272171s FAIL (2025-06-22 23:44:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.768298405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 35.355516544s FAIL (2025-06-22 23:43:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.355611453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X............................X.......X.................] duration: 27.226318552s FAIL (2025-06-22 23:43:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.226352362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 25.619253923s FAIL (2025-06-22 23:42:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.619277656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 40.839422537s FAIL (2025-06-22 23:42:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.839442793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X............................X.......X................] duration: 29.165095521s FAIL (2025-06-22 23:41:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.165122147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 26.142694273s FAIL (2025-06-22 23:41:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.142725314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X............................X.......X...............] duration: 35.625623556s FAIL (2025-06-22 23:41:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.625651804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 26.603485291s FAIL (2025-06-22 23:40:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.603518731s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X............................X.......X..............] duration: 19.034189659s FAIL (2025-06-22 23:39:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.034221376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 29.394515659s FAIL (2025-06-22 23:39:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.394544348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X............................X.......X.............] duration: 12.878552145s FAIL (2025-06-22 23:39:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.87857863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X............................X.......X............] duration: 28.057801203s FAIL (2025-06-22 23:38:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.057822425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 11.689971632s FAIL (2025-06-22 23:38:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.690017251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X............................X.......X...........] duration: 30.818642034s FAIL (2025-06-22 23:38:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.818682109s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 20.843289887s FAIL (2025-06-22 23:38:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.843314794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 42.229220738s FAIL (2025-06-22 23:37:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.229250443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X............................X.......X..........] duration: 13.028042369s FAIL (2025-06-22 23:37:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.02806728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X............................X.......X.........] duration: 14.332510472s FAIL (2025-06-22 23:36:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.332533878s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 24.30715522s FAIL (2025-06-22 23:36:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.307186003s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 51.425708807s FAIL (2025-06-22 23:35:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.425797502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X............................X.......X........] duration: 22.427806706s FAIL (2025-06-22 23:35:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.42784101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 20.705430455s FAIL (2025-06-22 23:34:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.705455854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X............................X.......X.......] duration: 15.397914083s FAIL (2025-06-22 23:34:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.397996789s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 26.025235276s FAIL (2025-06-22 23:34:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.025282214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X...............X............................X.......X......] duration: 23.258148462s FAIL (2025-06-22 23:33:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.258177467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 11.108762044s FAIL (2025-06-22 23:33:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.108814034s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 41.577408667s FAIL (2025-06-22 23:32:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.577444328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X...............X............................X.......X.....] duration: 28.761031861s FAIL (2025-06-22 23:32:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.761057503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X...............X............................X.......X....] duration: 16.728850032s FAIL (2025-06-22 23:31:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.728890458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X...............X............................X.......X...] duration: 18.141750689s FAIL (2025-06-22 23:31:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.141863725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.......................................X...................] duration: 16.828413676s FAIL (2025-06-22 23:30:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.828500457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X...............X............................X.......X..] duration: 49.326330115s FAIL (2025-06-22 23:30:39 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 49.326373185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................................X..................] duration: 10.362340636s FAIL (2025-06-22 23:30:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.362364957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................X.................] duration: 29.619867048s FAIL (2025-06-22 23:29:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.619893082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X...............X............................X.......X.] duration: 21.034622594s FAIL (2025-06-22 23:29:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.034663152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X...............X............................X.......X] duration: 1m0.025425335s FAIL (2025-06-22 23:29:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.025470691s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................X................] duration: 21.330167035s FAIL (2025-06-22 23:28:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.330195458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X.......................................X...............] duration: 22.062969363s FAIL (2025-06-22 23:27:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.062991634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X...............X............................X.......] duration: 31.117611016s FAIL (2025-06-22 23:27:31 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 31.117633911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......................................X..............] duration: 15.796553016s FAIL (2025-06-22 23:27:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.796574233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X...............X............................X......] duration: 33.405660449s FAIL (2025-06-22 23:27:00 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 33.405685903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X.......................................X.............] duration: 18.638183222s FAIL (2025-06-22 23:26:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.638209805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X...............X............................X.....] duration: 27.392766668s FAIL (2025-06-22 23:25:40 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 27.392787889s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......................................X............] duration: 13.219972569s FAIL (2025-06-22 23:25:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.220043516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X...............X............................X....] duration: 13.426042469s FAIL (2025-06-22 23:24:40 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== . err: <nil> Jun 22 23:21:01.905631330 (102h57m37.468954318s) [upload=82365]: Starting upload Jun 22 23:21:01.905710465 (102h57m37.469033535s) [upload=82365]: Done starting upload Jun 22 23:21:01.906002165 (102h57m37.469325169s) [upload=82365]: Uploading segments... Jun 22 23:21:01.906125019 (102h57m37.469448034s) [upload=82365]: Got next segment. Inline: false Jun 22 23:21:01.906233201 (102h57m37.469556211s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:21:01.906240171 (102h57m37.469563179s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:21:01.941180839 (102h57m37.50450385s) [seg_pos={0 0}, upload=82365]: Begin upload segment... Jun 22 23:21:01.941264896 (102h57m37.5045879s) [seg_pos={0 0}, upload=82365]: Done begin upload segment. Jun 22 23:21:01.941308677 (102h57m37.504631687s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941344040 (102h57m37.504667049s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941893263 (102h57m37.505216276s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.941932438 (102h57m37.505255442s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942585002 (102h57m37.505908014s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.942603141 (102h57m37.505926142s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943301900 (102h57m37.506624906s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.943888695 (102h57m37.507211706s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944260318 (102h57m37.50758333s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944400417 (102h57m37.507723407s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.944782492 (102h57m37.508105503s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945257714 (102h57m37.508580727s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.945643892 (102h57m37.508966904s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946033337 (102h57m37.509356349s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946436194 (102h57m37.509759306s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.946875911 (102h57m37.510198918s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.947390559 (102h57m37.510713572s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:01.948548196 (102h57m37.511871195s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Uploading piece... Jun 22 23:21:02.068774989 (102h57m37.632098036s) [upload=82365]: Next returned nil segment Jun 22 23:21:02.068797238 (102h57m37.632120253s) [upload=82365]: Waiting for error group managing segments... Jun 22 23:21:02.076863666 (102h57m37.640186676s) [noise=true , address=79.127.220.98:4421 , piece_id=2SUO5FCSHUDLBL25T65ZB7HGPW7VE53I6HIF3F7PUZNOCHAXLMMA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1p8pFsgRZqMhxU2mjjtwzArKbKGFZkosP4EmLCyz7RsZcgqw7e, address:79.127.220.98:4421): protocol: storage node overloaded, request limit: 500 Jun 22 23:21:02.203149091 (102h57m37.766472102s) [noise=true , address=104.36.82.7:5942 , piece_id=UIX72XSNLVYPBRD3DCAFRVQOCYPYJJLQSGJVVWX3CPCMXMLNXIRA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:5942): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.363182239 (102h57m37.926505251s) [noise=true , address=104.36.82.20:6461 , piece_id=E6DOXZTH7QR2XSKOJ4YFGZWBWBSHWQJXRAGAIVQG46HCCHXIXFSA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:6461): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:02.390126941 (102h57m37.953449938s) [noise=true , address=77.90.31.14:4681 , piece_id=VU36MFTCXFHE7GTBC2SYPYO4T34DGCRHEW2HHZTL3QPBTSCFYC7A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:02.816982817 (102h57m38.380305809s) [noise=true , address=104.36.82.8:5996 , piece_id=X7AVCT5BZ3CYQRBITRWH7UE3FJJZ33KIDKGRD22KJKWVWYUU5WQQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:03.052036478 (102h57m38.615359486s) [noise=true , address=104.36.82.4:5830 , piece_id=66ME3PQ6NWMOIDBR5T7ITN3XFJVL2URX4T6U4D4TVJUOA5N6BXUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.025839668 (102h57m39.589162659s) [noise=true , address=79.127.219.47:4061 , piece_id=I5FITLR462SRXCWHG3LTYXAMOCDLNTLRDY7X3Y34O727RRY5VYZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.050510449 (102h57m39.613833438s) [noise=true , address=45.91.116.7:5019 , piece_id=XIVNDD43O36HIUPGJTYJMJ6LCGNQ526RLYICXZIWSSGRV5ADCEZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:04.592735898 (102h57m40.156058906s) [noise=true , address=77.90.31.19:4856 , piece_id=ZIZFM7V4LGQIO6LM2CLJ5QHXIKCS4G5ZQIM4TCNTL64JBEOIUBZQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: storage node overloaded, request limit: 2000 Jun 22 23:21:06.187207632 (102h57m41.750530623s) [noise=true , address=79.127.219.38:4031 , piece_id=MOXJ63UH6OCIQJLRFTFZO64EPUQBN23LWMZR2ZEHJJ7GTKL46GUA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.137298912 (102h57m42.700621912s) [noise=true , address=104.36.82.34:6546 , piece_id=PBGSJQECYSRCDRIOYCAHNUWYYM6JB4KXBE7PGYEG7GGH6WIUWIOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.482937792 (102h57m43.046260798s) [noise=true , address=104.36.82.42:6827 , piece_id=X5MZGK2THZPOL7YYJKADG4MCWKLSBTUCTL5D7UWRTTIHPZVFQLIQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:07.766036797 (102h57m43.329359797s) [noise=true , address=45.91.116.6:4999 , piece_id=BTQVWGC3SA3MFQAQPXQC7BWSUGO4EWWKKH35AHWTPAZXI7GYNJXQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:12.015450697 (102h57m47.578773687s) [noise=true , address=104.36.82.7:5932 , piece_id=6K6EAS3SFM267TWE7L3VHXIR77EEQDZYT45UGMAMBVXQII2QUXDA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:17.763890693 (102h57m53.327213683s) [noise=true , address=104.36.82.55:4202 , piece_id=6NI45IDMH7UHJWI47VNSXNQDQA4HHFSQ26KVX7HA2MVIUWBZXWBQ, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:33.820395138 (102h58m9.383718128s) [noise=true , address=104.36.82.53:4135 , piece_id=TD3PM677X4YA37PLUBXLZIIKDIQ34BCTS3YYQFXQNOTFTI6DLOOA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:34.453479369 (102h58m10.016802368s) [noise=true , address=104.36.82.12:6184 , piece_id=3FYQV7YK7BGBK53HHZLBK6CSWCOQNMSCRJ34DFLCT6JTBATT3TVA, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595685148 (102h58m35.159008152s) [noise=true , address=104.36.82.54:4162 , piece_id=AGQICMO3CX25U5XBMGUVGY64YJIGQBAFSLEJIFM5X674IZ4V7T2A, seg_pos={0 0}, upload=82365]: Done uploading piece. err: <nil> Jun 22 23:21:59.595722710 (102h58m35.159045713s) [seg_pos={0 0}, upload=82365]: Segment reached optimal threshold of 14 pieces. Jun 22 23:21:59.595837185 (102h58m35.159160188s) [seg_pos={0 0}, upload=82365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jun 22 23:21:59.595971260 (102h58m35.15929426s) [upload=82365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 22 23:21:59.595978993 (102h58m35.159301993s) [upload=82365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 23:21:59.644099523 (102h58m35.207422535s) [upload=82365]: Done uploading segments... Jun 22 23:21:59.644137178 (102h58m35.207460174s) [upload=82365]: Upload finished. err: <nil> Duration: 13.426069491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X.......................................X...........] duration: 20.924535333s FAIL (2025-06-22 23:24:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.924561173s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)