Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...........XX...X......X...X.........X............X........] duration: 10.156833386s FAIL (2025-07-09 01:09:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.156856399s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.....X...........XX...X.XXX...X.....XX..X....XX..X.......X..] duration: 48.620508833s FAIL (2025-07-09 01:09:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.62052622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X.......X.................................] duration: 514.234236ms FAIL (2025-07-09 01:09:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 514.26485ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X....X.......X................................] duration: 623.129838ms FAIL (2025-07-09 01:08:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 623.161447ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [......X...........XX...X.XXX...X.....XX..X....XX..X.......X.] duration: 50.677729556s FAIL (2025-07-09 01:08:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.677750307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...........XX...X......X...X.........X............X.......] duration: 3.762851419s FAIL (2025-07-09 01:08:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.762879246s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X....X.......X...............................] duration: 522.210418ms FAIL (2025-07-09 01:08:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 522.244299ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X....X.......X..............................] duration: 557.74183ms FAIL (2025-07-09 01:08:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 557.793461ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.......X...........XX...X.XXX...X.....XX..X....XX..X.......X] duration: 1m0.001051595s FAIL (2025-07-09 01:07:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001082406s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X...........XX...X......X...X.........X............X......] duration: 14.379982542s FAIL (2025-07-09 01:07:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.380004766s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X...........XX...X......X...X.........X............X.....] duration: 28.676360925s FAIL (2025-07-09 01:07:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.676399682s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X....X.......X.............................] duration: 598.017184ms FAIL (2025-07-09 01:06:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 598.070104ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [........X...........XX...X.XXX...X.....XX..X....XX..X.......] duration: 54.072734669s FAIL (2025-07-09 01:06:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.072765939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X....X.......X............................] duration: 392.743767ms FAIL (2025-07-09 01:06:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.771871ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...X...........XX...X......X...X.........X............X....] duration: 6.35733659s FAIL (2025-07-09 01:06:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.35742408s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X...X...........XX...X......X...X.........X............X...] duration: 21.575601266s FAIL (2025-07-09 01:05:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.575655752s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X....X.......X...........................] duration: 605.053152ms FAIL (2025-07-09 01:05:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 605.097259ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.........X...........XX...X.XXX...X.....XX..X....XX..X......] duration: 48.781030688s FAIL (2025-07-09 01:05:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.78104936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X...X...........XX...X......X...X.........X............X..] duration: 19.868603673s FAIL (2025-07-09 01:05:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.868641724s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..........X...........XX...X.XXX...X.....XX..X....XX..X.....] duration: 53.586226955s FAIL (2025-07-09 01:04:36 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.586242961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X...X...........XX...X......X...X.........X............X.] duration: 5.35544667s FAIL (2025-07-09 01:04:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.355460507s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X....X.......X..........................] duration: 691.523837ms FAIL (2025-07-09 01:04:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 691.578972ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X...X...X...........XX...X......X...X.........X............X] duration: 30.002016488s FAIL (2025-07-09 01:04:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002047672s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X...X...X...........XX...X......X...X.........X............] duration: 17.250898199s FAIL (2025-07-09 01:03:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.250922669s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...........X...........XX...X.XXX...X.....XX..X....XX..X....] duration: 49.084466661s FAIL (2025-07-09 01:03:43 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.084497047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X....X.......X.........................] duration: 446.333728ms FAIL (2025-07-09 01:03:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.361989ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X...X...X...........XX...X......X...X.........X...........] duration: 7.613541782s FAIL (2025-07-09 01:03:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.613563342s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X...X...X...........XX...X......X...X.........X..........] duration: 1.651389294s FAIL (2025-07-09 01:02:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.651414111s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [............X...........XX...X.XXX...X.....XX..X....XX..X...] duration: 44.542498469s FAIL (2025-07-09 01:02:53 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.542522501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X....X.......X........................] duration: 494.682398ms FAIL (2025-07-09 01:02:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.708426ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X...X...X...........XX...X......X...X.........X.........] duration: 8.302684234s FAIL (2025-07-09 01:02:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.302718968s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.............X...........XX...X.XXX...X.....XX..X....XX..X..] duration: 37.314646242s FAIL (2025-07-09 01:02:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.3146778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X....X.......X.......................] duration: 426.308271ms FAIL (2025-07-09 01:01:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 426.332041ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....X.......X......................] duration: 602.837835ms FAIL (2025-07-09 01:01:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 602.876021ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..............X...........XX...X.XXX...X.....XX..X....XX..X.] duration: 46.321608581s FAIL (2025-07-09 01:01:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.321627866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X...X...X...........XX...X......X...X.........X........] duration: 759.410339ms FAIL (2025-07-09 01:00:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 759.451116ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....X.......X.....................] duration: 995.193779ms FAIL (2025-07-09 01:00:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 995.318245ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...............X...........XX...X.XXX...X.....XX..X....XX..X] duration: 1m0.001499359s FAIL (2025-07-09 01:00:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001529499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X...X...X...........XX...X......X...X.........X.......] duration: 780.843786ms FAIL (2025-07-09 00:59:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 780.876546ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....X.......X....................] duration: 444.93962ms FAIL (2025-07-09 00:59:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.965674ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X...X...X...........XX...X......X...X.........X......] duration: 22.414526015s FAIL (2025-07-09 00:59:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.414569194s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [................X...........XX...X.XXX...X.....XX..X....XX..] duration: 39.988350298s FAIL (2025-07-09 00:58:46 UTC) context deadline exceeded Duration: 39.98840143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....X.......X...................] duration: 461.785597ms FAIL (2025-07-09 00:58:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.820997ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.......X...X...X...........XX...X......X...X.........X.....] duration: 8.857239875s FAIL (2025-07-09 00:58:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.85726507s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X................X...........XX...X.XXX...X.....XX..X....XX.] duration: 41.435483526s FAIL (2025-07-09 00:58:05 UTC) context deadline exceeded Duration: 41.435502604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....X.......X..................] duration: 881.048908ms FAIL (2025-07-09 00:57:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 881.076425ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.......X...X...X...........XX...X......X...X.........X....] duration: 10.620038092s FAIL (2025-07-09 00:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.620062893s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X................X...........XX...X.XXX...X.....XX..X....XX] duration: 1m0.001037538s FAIL (2025-07-09 00:57:24 UTC) context deadline exceeded Duration: 1m0.001054564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....X.......X.................] duration: 457.9299ms FAIL (2025-07-09 00:57:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.959637ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....X.......X................] duration: 12.255418957s FAIL (2025-07-09 00:56:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.255455738s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)