Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.X........................X..] duration: 16.815049532s FAIL (2025-06-23 00:22:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.815102029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.X........................X.] duration: 16.363606824s FAIL (2025-06-23 00:21:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.363859274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................X.X........................X] duration: 1m0.000549152s FAIL (2025-06-23 00:21:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000571838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 13.75290145s FAIL (2025-06-23 00:19:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.752921788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 11.713063307s FAIL (2025-06-23 00:18:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.713076249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 26.261546729s FAIL (2025-06-23 00:18:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.261571904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 22.667144333s FAIL (2025-06-23 00:17:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.667547417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 22.47873791s FAIL (2025-06-23 00:16:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.47876528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 18.412483171s FAIL (2025-06-23 00:16:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.412499533s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 40.788948196s FAIL (2025-06-23 00:16:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.788968991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 17.59302903s FAIL (2025-06-23 00:15:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.593060619s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 29.329290888s FAIL (2025-06-23 00:15:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.329312412s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 18.915789466s FAIL (2025-06-23 00:15:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.915822029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 56.704496627s FAIL (2025-06-23 00:14:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.704515357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 18.69540617s FAIL (2025-06-23 00:14:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.69543431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 13.229199258s FAIL (2025-06-23 00:13:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.229223393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 56.586030872s FAIL (2025-06-23 00:13:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.586045521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 23.371223591s FAIL (2025-06-23 00:13:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.371251966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 34.04843644s FAIL (2025-06-23 00:12:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.048464599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 24.651464247s FAIL (2025-06-23 00:12:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.65149771s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 16.410149102s FAIL (2025-06-23 00:12:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.410169189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 28.223337466s FAIL (2025-06-23 00:11:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.2233624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.......X...................................................] duration: 31.688112478s FAIL (2025-06-23 00:11:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.688164674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 21.525217757s FAIL (2025-06-23 00:10:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.525343139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 5.730439954s FAIL (2025-06-23 00:10:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.730475252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X.......X..................................................] duration: 11.099484397s FAIL (2025-06-23 00:09:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.099524493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 21.674718852s FAIL (2025-06-23 00:09:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.674739245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X.......X.................................................] duration: 22.907709686s FAIL (2025-06-23 00:09:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.907739272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.......X................................................] duration: 36.662869566s FAIL (2025-06-23 00:09:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 36.662894174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 21.502894408s FAIL (2025-06-23 00:09:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.502926763s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 5.098983605s FAIL (2025-06-23 00:08:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.099008251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 36.189833506s FAIL (2025-06-23 00:08:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.189859294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X.......X...............................................] duration: 7.275920913s FAIL (2025-06-23 00:07:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.275958794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......X..............................................] duration: 8.873728998s FAIL (2025-06-23 00:06:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.873756701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 5.197137663s FAIL (2025-06-23 00:06:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.197172929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X.......X.............................................] duration: 34.47059757s FAIL (2025-06-23 00:06:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.470628274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 22.909905355s FAIL (2025-06-23 00:06:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.910000611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......X............................................] duration: 20.01272616s FAIL (2025-06-23 00:05:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.012785554s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 15.883597689s FAIL (2025-06-23 00:05:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.88365465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 28.415633712s FAIL (2025-06-23 00:04:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.415661119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X.......X...........................................] duration: 21.288041562s FAIL (2025-06-23 00:04:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.288073729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......X..........................................] duration: 7.977896553s FAIL (2025-06-23 00:03:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.977929313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 8.438911713s FAIL (2025-06-23 00:03:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.438932617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......X.........................................] duration: 23.749648786s FAIL (2025-06-23 00:02:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.749679374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 22.039359699s FAIL (2025-06-23 00:02:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.039384177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.......................................................X.X.] duration: 24.7169s FAIL (2025-06-23 00:02:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.716926051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......X........................................] duration: 20.356294083s FAIL (2025-06-23 00:01:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.356359711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.......................................................X.X] duration: 1m0.005268062s FAIL (2025-06-23 00:01:34 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.005292619s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................................X.] duration: 15.989740423s FAIL (2025-06-23 00:00:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.989757722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X.......X.......................................] duration: 18.885522101s FAIL (2025-06-23 00:00:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 868s) [upload=82458]: Starting upload Jun 22 23:28:10.142414098 (103h4m45.705737101s) [upload=82458]: Done starting upload Jun 22 23:28:10.142493502 (103h4m45.705816502s) [upload=82458]: Uploading segments... Jun 22 23:28:10.142834706 (103h4m45.706157705s) [upload=82458]: Got next segment. Inline: false Jun 22 23:28:10.142911185 (103h4m45.706234188s) [upload=82458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 23:28:10.142920605 (103h4m45.706243606s) [upload=82458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 23:28:10.177212013 (103h4m45.740535018s) [seg_pos={0 0}, upload=82458]: Begin upload segment... Jun 22 23:28:10.177287365 (103h4m45.740610355s) [seg_pos={0 0}, upload=82458]: Done begin upload segment. Jun 22 23:28:10.177332260 (103h4m45.740655262s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.177827557 (103h4m45.741150566s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178033043 (103h4m45.741356044s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.178803820 (103h4m45.742126825s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179059054 (103h4m45.742382053s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179807050 (103h4m45.743130059s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.179864741 (103h4m45.743187752s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180344625 (103h4m45.743667635s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.180763034 (103h4m45.744086042s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.183550521 (103h4m45.746873531s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.187243851 (103h4m45.750566855s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188168229 (103h4m45.751491233s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.188988263 (103h4m45.752311264s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.189795910 (103h4m45.753118911s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.190218441 (103h4m45.753541444s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.195932207 (103h4m45.759255201s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.204524067 (103h4m45.767847053s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.206927735 (103h4m45.770250722s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Uploading piece... Jun 22 23:28:10.252642050 (103h4m45.815965035s) [upload=82458]: Next returned nil segment Jun 22 23:28:10.252659523 (103h4m45.815982508s) [upload=82458]: Waiting for error group managing segments... Jun 22 23:28:10.274998542 (103h4m45.838321546s) [noise=true , address=79.127.219.38:4007 , piece_id=D224T3AAW2ONZ3GWSKDT63A72VOYLWBHBXYFRJHU6YH52ODIW36A, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.275674802 (103h4m45.838997803s) [noise=true , address=79.127.219.51:4194 , piece_id=GZPYLCTT6DHHJHNYSPPA6AHP4FMVOIQD3CWKDQFUCA4P6INBOAHA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12kN12Zuvae8zYxxpNSTiHeznoeysHx8xGwafUR8N92ktKJwfEK, address:79.127.219.51:4194): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:10.284572669 (103h4m45.847895675s) [noise=true , address=79.127.220.98:4410 , piece_id=QGTYNM63UTVQHWLE24OR52FFPL4NI7PPWIHTSWATAFWTKYAD6IRQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 22 23:28:10.285316354 (103h4m45.848639344s) [noise=true , address=45.91.116.14:5206 , piece_id=BRAI7AYRCA224HXDHPLYCGC4QUDPEGRDFTLO6LQUHRXZC4MJ7ASA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.316826961 (103h4m45.880149967s) [noise=true , address=77.90.31.14:4678 , piece_id=K7QKRW3HIALKDAUCEG3MTOOMGB334E3RW57GTQ5GIHZUBVPVKOQA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:10.666944704 (103h4m46.230267713s) [noise=true , address=104.36.82.12:6180 , piece_id=H7A3N642Q5UJU5D4SWJXSKPZGUMOKWKDVKPQGKHP4RGIVHAPVSWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:6180): protocol: storage node overloaded, request limit: 2000 Jun 22 23:28:11.237437858 (103h4m46.800760843s) [noise=true , address=104.36.82.13:6216 , piece_id=VZJFEPNAIFH7NPJUOM3XQJRNQLKR2OAPTUMWAKABJBFCO7VNSB7Q, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.218225199 (103h4m47.781548191s) [noise=true , address=45.91.116.6:4994 , piece_id=OQAXIIBSICVUSMXI5P7YMGKDCJF64Z5CG3YABQ7LGXQPSAGKMASQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:12.697707912 (103h4m48.261030897s) [noise=true , address=104.36.82.15:6287 , piece_id=ZJ7ZYFPX4DIEFZP6Q2OJ73FW6GNHQ54DFPAARDV33MP6JAHLFORA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:16.941609598 (103h4m52.504932585s) [noise=true , address=104.36.82.53:4146 , piece_id=HLCXRP66YUXJ5KPVMCOQRMBTHNQ57LVSIZC3CL4F6JAAVE4EGPPQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:17.823800488 (103h4m53.387123481s) [noise=true , address=104.36.82.4:5809 , piece_id=U7UUUKKJ3BXIHPITU3LGCTRFOR5J4ZBOV47GV366H4HWXMZMCLHQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:18.539391528 (103h4m54.10271452s) [noise=true , address=104.36.82.10:6087 , piece_id=H27JOFYPG44ZZZIZETZI2X6CJD7UMPOGDC4KE5STC3NISRY3GHWA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:19.007174456 (103h4m54.570497453s) [noise=true , address=77.90.31.10:4540 , piece_id=E3OTRZIJTLC7IN3OJLZWXKEEXX4SUSINBYDCOABBXY4N7Z26SCTQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:25.067238481 (103h5m0.630561472s) [noise=true , address=104.36.82.52:4110 , piece_id=ARGQ4ATAAQ4WC4STRNOZTS2OGRHA5XMKK6B4CBGY4HJ6U67PFDEA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:29.354837739 (103h5m4.918160728s) [noise=true , address=104.36.82.4:5808 , piece_id=SMHIEL72R7A2ITBI2FEY25IRM45PWHQC33JGA4EHSZOBL6LGGQSQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:28:31.726250308 (103h5m7.289573301s) [noise=true , address=104.36.82.3:5779 , piece_id=4JILIGN2ZMDKFTLIQK56YM5FRMX6VTPJMM25XL3VXEGPFAZZWDZA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: <nil> Jun 22 23:29:10.101364526 (103h5m45.664687522s) [noise=true , address=104.36.82.56:4260 , piece_id=52TBM6WZQRRZX4EULBKLVIJEHP2IXEIMYZH3FLB5GKR3ILBJIXCA, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39880->104.36.82.56:4260: use of closed network connection Jun 22 23:29:10.101405169 (103h5m45.664728159s) [noise=true , address=104.36.82.5:5847 , piece_id=5Y3IDUDJSRZIW2VYJLL3LCU7KHKULUEZAIJB65OWXUEPWEZ62DGQ, seg_pos={0 0}, upload=82458]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:5847): protocol: context deadline exceeded; context deadline exceeded Jun 22 23:29:10.101451900 (103h5m45.664774902s) [seg_pos={0 0}, upload=82458]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 23:29:10.124432723 (103h5m45.687755717s) [upload=82458]: Done uploading segments... Jun 22 23:29:10.124456529 (103h5m45.687779526s) [upload=82458]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.885554929s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)