Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X......X...X.........X............X.........XXX........X..X] duration: 30.001433585s FAIL (2025-07-09 01:22:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001452331s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 449.21293ms FAIL (2025-07-09 01:22:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.248039ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [....XX...X.XXX...X.....XX..X....XX..X.......X....XX..X..XX..] duration: 55.275529105s FAIL (2025-07-09 01:22:23 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 55.275567261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 451.52967ms FAIL (2025-07-09 01:22:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.560733ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 436.081629ms FAIL (2025-07-09 01:21:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.111915ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X......X...X.........X............X.........XXX........X..] duration: 499.519482ms FAIL (2025-07-09 01:21:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 499.585433ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X......X...X.........X............X.........XXX........X.] duration: 12.803870661s FAIL (2025-07-09 01:21:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.803888047s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.....XX...X.XXX...X.....XX..X....XX..X.......X....XX..X..XX.] duration: 26.454792292s FAIL (2025-07-09 01:21:00 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.454814422s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [......XX...X.XXX...X.....XX..X....XX..X.......X....XX..X..XX] duration: 1m0.0020183s FAIL (2025-07-09 01:20:34 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002031586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......X...................................................] duration: 583.581301ms FAIL (2025-07-09 01:20:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 583.608273ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X...X......X...X.........X............X.........XXX........X] duration: 30.00069758s FAIL (2025-07-09 01:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000723352s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......X..................................................] duration: 494.871266ms FAIL (2025-07-09 01:20:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.903426ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......X.................................................] duration: 385.061245ms FAIL (2025-07-09 01:19:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.088898ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX...X......X...X.........X............X.........XXX........] duration: 25.498226237s FAIL (2025-07-09 01:19:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.498249112s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......XX...X.XXX...X.....XX..X....XX..X.......X....XX..X..X] duration: 1m0.00237711s FAIL (2025-07-09 01:19:34 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002399486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......X................................................] duration: 410.847991ms FAIL (2025-07-09 01:19:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.893782ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX...X......X...X.........X............X.........XXX.......] duration: 4.483115948s FAIL (2025-07-09 01:18:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.48314011s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.......X...............................................] duration: 549.344939ms FAIL (2025-07-09 01:18:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 549.374973ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [........XX...X.XXX...X.....XX..X....XX..X.......X....XX..X..] duration: 46.278855526s FAIL (2025-07-09 01:18:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.278883022s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X....X.......X..............................................] duration: 632.846943ms FAIL (2025-07-09 01:18:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 632.894867ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XX...X......X...X.........X............X.........XXX......] duration: 10.047651438s FAIL (2025-07-09 01:17:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.047680253s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....X.......X.............................................] duration: 420.10623ms FAIL (2025-07-09 01:17:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.129705ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....X.......X............................................] duration: 437.443558ms FAIL (2025-07-09 01:17:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.48412ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.........XX...X.XXX...X.....XX..X....XX..X.......X....XX..X.] duration: 48.788571408s FAIL (2025-07-09 01:17:02 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.788667771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....X.......X...........................................] duration: 485.311151ms FAIL (2025-07-09 01:16:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 485.339522ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..........XX...X.XXX...X.....XX..X....XX..X.......X....XX..X] duration: 1m0.000974748s FAIL (2025-07-09 01:16:12 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001001614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...XX...X......X...X.........X............X.........XXX.....] duration: 969.754367ms FAIL (2025-07-09 01:16:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 969.78135ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....X.......X..........................................] duration: 7.979451367s FAIL (2025-07-09 01:15:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.979482401s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....XX...X......X...X.........X............X.........XXX....] duration: 9.492930003s FAIL (2025-07-09 01:15:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.492955154s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X....X.......X.........................................] duration: 442.45239ms FAIL (2025-07-09 01:15:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.485373ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [...........XX...X.XXX...X.....XX..X....XX..X.......X....XX..] duration: 43.867126095s FAIL (2025-07-09 01:14:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.867141769s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....XX...X......X...X.........X............X.........XXX...] duration: 491.191714ms FAIL (2025-07-09 01:14:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.209745ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......XX...X......X...X.........X............X.........XXX..] duration: 22.847928693s FAIL (2025-07-09 01:14:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.847975549s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X....X.......X........................................] duration: 413.364035ms FAIL (2025-07-09 01:14:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.392229ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X...........XX...X.XXX...X.....XX..X....XX..X.......X....XX.] duration: 44.835013879s FAIL (2025-07-09 01:14:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.835024667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X...........XX...X.XXX...X.....XX..X....XX..X.......X....XX] duration: 1m0.000506391s FAIL (2025-07-09 01:13:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000516329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X....X.......X.......................................] duration: 392.197732ms FAIL (2025-07-09 01:13:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.230104ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X....X.......X......................................] duration: 986.314299ms FAIL (2025-07-09 01:13:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 986.417185ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......XX...X......X...X.........X............X.........XXX.] duration: 3.05631577s FAIL (2025-07-09 01:12:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.056334745s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........XX...X......X...X.........X............X.........XXX] duration: 30.000864054s FAIL (2025-07-09 01:12:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000887289s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [..X...........XX...X.XXX...X.....XX..X....XX..X.......X....X] duration: 1m0.001374474s FAIL (2025-07-09 01:12:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001400241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X....X.......X.....................................] duration: 491.330549ms FAIL (2025-07-09 01:12:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.361164ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........XX...X......X...X.........X............X.........XX] duration: 30.000808599s FAIL (2025-07-09 01:11:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000831159s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...X...........XX...X.XXX...X.....XX..X....XX..X.......X....] duration: 51.905034248s FAIL (2025-07-09 01:11:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.90505192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........XX...X......X...X.........X............X.........X] duration: 30.001055421s FAIL (2025-07-09 01:11:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001077728s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X....X.......X....................................] duration: 516.53995ms FAIL (2025-07-09 01:10:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 516.569649ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [....X...........XX...X.XXX...X.....XX..X....XX..X.......X...] duration: 53.743366194s FAIL (2025-07-09 01:10:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.743383933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........XX...X......X...X.........X............X.........] duration: 4.143953164s FAIL (2025-07-09 01:10:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.143973882s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X.......X...................................] duration: 562.515868ms FAIL (2025-07-09 01:10:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 562.592302ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X.......X..................................] duration: 5.738105075s FAIL (2025-07-09 01:10:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.738145059s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)