Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................................X] duration: 1m0.009958626s FAIL (2025-06-23 00:00:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.009989682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X.......X......................................] duration: 20.40073693s FAIL (2025-06-22 23:59:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.400806873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 23.660361579s FAIL (2025-06-22 23:59:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.660386076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X.......X.....................................] duration: 29.914011722s FAIL (2025-06-22 23:58:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.914033109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......X....................................] duration: 30.469017129s FAIL (2025-06-22 23:58:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.469048014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 20.657771264s FAIL (2025-06-22 23:58:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.65780421s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 14.452936613s FAIL (2025-06-22 23:57:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.452960561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X.......X...................................] duration: 26.746324622s FAIL (2025-06-22 23:56:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.746352221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 12.515345937s FAIL (2025-06-22 23:56:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.515375071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 20.73298858s FAIL (2025-06-22 23:56:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.733018309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X.......X..................................] duration: 20.365864298s FAIL (2025-06-22 23:55:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.365891383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 48.069424501s FAIL (2025-06-22 23:55:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.069446633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X.......X.................................] duration: 29.391937561s FAIL (2025-06-22 23:55:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.392045143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 21.198655916s FAIL (2025-06-22 23:54:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.198692325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X.......X................................] duration: 29.511822383s FAIL (2025-06-22 23:54:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.511847519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X.......X...............................] duration: 32.613520114s FAIL (2025-06-22 23:54:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.613545498s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 21.50597341s FAIL (2025-06-22 23:54:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.506056726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.......X..............................] duration: 34.107859262s FAIL (2025-06-22 23:53:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.107908548s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 17.921473215s FAIL (2025-06-22 23:53:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.921504313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 14.703351891s FAIL (2025-06-22 23:52:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.70343512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X.......X.............................] duration: 26.775887901s FAIL (2025-06-22 23:52:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.775916164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 22.471977795s FAIL (2025-06-22 23:51:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.472032055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.......X............................] duration: 19.080849669s FAIL (2025-06-22 23:50:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.080876813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 16.096971379s FAIL (2025-06-22 23:50:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.096994018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 22.01524238s FAIL (2025-06-22 23:50:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.015390354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 53.041569498s FAIL (2025-06-22 23:49:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.041647856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X.......X...........................] duration: 21.059171264s FAIL (2025-06-22 23:49:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.059203511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.......X..........................] duration: 30.894131953s FAIL (2025-06-22 23:49:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.894160399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 24.704786327s FAIL (2025-06-22 23:48:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.70481648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 29.615344564s FAIL (2025-06-22 23:48:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.615370316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.......X.........................] duration: 26.111351958s FAIL (2025-06-22 23:48:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.111384381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 20.33476693s FAIL (2025-06-22 23:47:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.334786887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.......X........................] duration: 11.156163849s FAIL (2025-06-22 23:47:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.156194902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 19.568981678s FAIL (2025-06-22 23:46:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.569002685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X.......X.......................] duration: 19.12830245s FAIL (2025-06-22 23:46:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.128347516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X............................X.......X......................] duration: 10.993847869s FAIL (2025-06-22 23:45:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.993880124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 16.220240374s FAIL (2025-06-22 23:45:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.220270003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X............................X.......X.....................] duration: 10.824308444s FAIL (2025-06-22 23:45:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.824336211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X............................X.......X....................] duration: 26.590238655s FAIL (2025-06-22 23:45:02 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.590320006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 14.85730947s FAIL (2025-06-22 23:44:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.857397642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X............................X.......X...................] duration: 13.477114478s FAIL (2025-06-22 23:44:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.477146019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 27.151515038s FAIL (2025-06-22 23:44:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.151554692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X............................X.......X..................] duration: 19.768272171s FAIL (2025-06-22 23:44:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.768298405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 35.355516544s FAIL (2025-06-22 23:43:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.355611453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X............................X.......X.................] duration: 27.226318552s FAIL (2025-06-22 23:43:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.226352362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 25.619253923s FAIL (2025-06-22 23:42:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.619277656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 40.839422537s FAIL (2025-06-22 23:42:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.839442793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X............................X.......X................] duration: 29.165095521s FAIL (2025-06-22 23:41:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.165122147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 26.142694273s FAIL (2025-06-22 23:41:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.142725314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X............................X.......X...............] duration: 35.625623556s FAIL (2025-06-22 23:41:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.625651804s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)