Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X...X...X...........XX...X......X...X.........X......] duration: 22.414526015s FAIL (2025-07-09 00:59:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.414569194s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [................X...........XX...X.XXX...X.....XX..X....XX..] duration: 39.988350298s FAIL (2025-07-09 00:58:46 UTC) context deadline exceeded Duration: 39.98840143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....X.......X...................] duration: 461.785597ms FAIL (2025-07-09 00:58:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.820997ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.......X...X...X...........XX...X......X...X.........X.....] duration: 8.857239875s FAIL (2025-07-09 00:58:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.85726507s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X................X...........XX...X.XXX...X.....XX..X....XX.] duration: 41.435483526s FAIL (2025-07-09 00:58:05 UTC) context deadline exceeded Duration: 41.435502604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....X.......X..................] duration: 881.048908ms FAIL (2025-07-09 00:57:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 881.076425ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.......X...X...X...........XX...X......X...X.........X....] duration: 10.620038092s FAIL (2025-07-09 00:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.620062893s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X................X...........XX...X.XXX...X.....XX..X....XX] duration: 1m0.001037538s FAIL (2025-07-09 00:57:24 UTC) context deadline exceeded Duration: 1m0.001054564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....X.......X.................] duration: 457.9299ms FAIL (2025-07-09 00:57:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.959637ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....X.......X................] duration: 12.255418957s FAIL (2025-07-09 00:56:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.255455738s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.......X...X...X...........XX...X......X...X.........X...] duration: 559.086894ms FAIL (2025-07-09 00:56:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 559.119465ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X....X.......X...............] duration: 771.661027ms FAIL (2025-07-09 00:56:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 771.68646ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..X................X...........XX...X.XXX...X.....XX..X....X] duration: 1m0.001658806s FAIL (2025-07-09 00:56:24 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001675373s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.......X...X...X...........XX...X......X...X.........X..] duration: 18.563878775s FAIL (2025-07-09 00:56:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.563923152s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X....X.......X..............] duration: 7.154413009s FAIL (2025-07-09 00:55:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.154437684s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X.......X...X...X...........XX...X......X...X.........X.] duration: 28.63271994s FAIL (2025-07-09 00:55:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.632746921s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...X................X...........XX...X.XXX...X.....XX..X....] duration: 58.41871218s FAIL (2025-07-09 00:55:24 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 58.418732707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X.......X...X...X...........XX...X......X...X.........X] duration: 30.000618306s FAIL (2025-07-09 00:55:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000639617s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X....X.......X.............] duration: 682.653831ms FAIL (2025-07-09 00:54:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 682.69567ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X.......X...X...X...........XX...X......X...X.........] duration: 14.972383112s FAIL (2025-07-09 00:54:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.972412872s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X....X.......X............] duration: 592.712858ms FAIL (2025-07-09 00:54:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 592.740999ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X...X................X...........XX...X.XXX...X.....XX..X...] duration: 52.418544491s FAIL (2025-07-09 00:54:25 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.418567566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X......X.......X...X...X...........XX...X......X...X........] duration: 9.764968445s FAIL (2025-07-09 00:54:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.765002895s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X......X.......X...X...X...........XX...X......X...X.......] duration: 20.014018636s FAIL (2025-07-09 00:53:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.014048358s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X...X................X...........XX...X.XXX...X.....XX..X..] duration: 54.875471621s FAIL (2025-07-09 00:53:24 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.875488777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X....X.......X...........] duration: 811.490614ms FAIL (2025-07-09 00:53:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 811.583308ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X....X.......X..........] duration: 866.809188ms FAIL (2025-07-09 00:52:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 866.838287ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..X...X................X...........XX...X.XXX...X.....XX..X.] duration: 56.314791197s FAIL (2025-07-09 00:52:29 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.314802891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X......X.......X...X...X...........XX...X......X...X......] duration: 1.611583515s FAIL (2025-07-09 00:52:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.611602136s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X....X.......X.........] duration: 12.56308447s FAIL (2025-07-09 00:51:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.563122997s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...X...X................X...........XX...X.XXX...X.....XX..X] duration: 1m0.000504136s FAIL (2025-07-09 00:51:33 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000518872s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X......X.......X...X...X...........XX...X......X...X.....] duration: 504.473494ms FAIL (2025-07-09 00:51:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.493285ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X...X......X.......X...X...X...........XX...X......X...X....] duration: 14.173590724s FAIL (2025-07-09 00:51:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.173612676s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X....X.......X........] duration: 458.697496ms FAIL (2025-07-09 00:50:34 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.730623ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [....X...X................X...........XX...X.XXX...X.....XX..] duration: 43.867177269s FAIL (2025-07-09 00:50:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.867197952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X....X.......X.......] duration: 23.124320229s FAIL (2025-07-09 00:50:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 23.124352016s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X...X......X.......X...X...X...........XX...X......X...X...] duration: 649.793464ms FAIL (2025-07-09 00:50:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 649.909951ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.....X...X................X...........XX...X.XXX...X.....XX.] duration: 56.85763401s FAIL (2025-07-09 00:49:43 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.857649864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X...X......X.......X...X...X...........XX...X......X...X..] duration: 9.200925958s FAIL (2025-07-09 00:49:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.200960919s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X...X......X.......X...X...X...........XX...X......X...X.] duration: 27.662107148s FAIL (2025-07-09 00:49:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.662136919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X....X.......X......] duration: 553.338916ms FAIL (2025-07-09 00:49:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 553.36827ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [......X...X................X...........XX...X.XXX...X.....XX] duration: 1m0.002707767s FAIL (2025-07-09 00:48:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002719485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X...X......X.......X...X...X...........XX...X......X...X] duration: 30.00093762s FAIL (2025-07-09 00:48:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00120027s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X....X.......X.....] duration: 574.696429ms FAIL (2025-07-09 00:47:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 574.72295ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.......X...X................X...........XX...X.XXX...X.....X] duration: 1m0.001032401s FAIL (2025-07-09 00:47:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001051076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X....X.......X....] duration: 11.914698461s FAIL (2025-07-09 00:47:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.91472029s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X...X......X.......X...X...X...........XX...X......X...] duration: 29.323780544s FAIL (2025-07-09 00:47:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.323805708s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X....X.......X...] duration: 29.799855387s FAIL (2025-07-09 00:47:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 29.799891806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X....X.......X..] duration: 13.21083714s FAIL (2025-07-09 00:46:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== errs: <nil> Jul 9 00:43:45.211297211 (375h32m12.137591901s) [upload=300745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:43:45.211302471 (375h32m12.137597162s) [upload=300745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:43:45.269341492 (375h32m12.195636178s) [upload=300745]: Done uploading segments... Jul 9 00:43:45.269358444 (375h32m12.19565313s) [upload=300745]: Upload finished. err: <nil> Jul 9 00:44:41.129146431 (375h33m8.055441118s) [upload=300759]: Starting upload Jul 9 00:44:41.129253693 (375h33m8.055548379s) [upload=300759]: Done starting upload Jul 9 00:44:41.129403652 (375h33m8.05569833s) [upload=300759]: Uploading segments... Jul 9 00:44:41.129792975 (375h33m8.056087669s) [upload=300759]: Got next segment. Inline: false Jul 9 00:44:41.129845465 (375h33m8.056140152s) [upload=300759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:44:41.129851436 (375h33m8.056146123s) [upload=300759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:44:41.161096648 (375h33m8.087391325s) [seg_pos={0 0}, upload=300759]: Begin upload segment... Jul 9 00:44:41.161143590 (375h33m8.087438271s) [seg_pos={0 0}, upload=300759]: Done begin upload segment. Jul 9 00:44:41.161167424 (375h33m8.087462113s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161202049 (375h33m8.087496745s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161568494 (375h33m8.087863181s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161739184 (375h33m8.08803387s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.161926542 (375h33m8.08822123s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162264603 (375h33m8.088559289s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162275501 (375h33m8.088570197s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.162710355 (375h33m8.089005047s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163083824 (375h33m8.089378519s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163449862 (375h33m8.089744559s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163765583 (375h33m8.090060273s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.163963749 (375h33m8.090258446s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164440292 (375h33m8.090734988s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.164860336 (375h33m8.091155033s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165178752 (375h33m8.091473449s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.165665625 (375h33m8.09196032s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Uploading piece... Jul 9 00:44:41.266729879 (375h33m8.193024572s) [upload=300759]: Next returned nil segment Jul 9 00:44:41.266753705 (375h33m8.193048411s) [upload=300759]: Waiting for error group managing segments... Jul 9 00:44:41.272587189 (375h33m8.198881879s) [noise=true , address=45.91.116.11:4117 , piece_id=W7YOBPGYU72MMDP6QZYVKV2344UGEO4TYQPZOIQ7FDMMYEIQ6I5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273314775 (375h33m8.199609455s) [noise=true , address=77.90.31.12:4590 , piece_id=UOIGX7N2FC2SBE2RQIFO63XYXKVHSIHOZ5VWJPJSPRU74RYW7B4A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.273750474 (375h33m8.200045167s) [noise=true , address=45.91.116.7:4010 , piece_id=ZA27QVAEZEUUSEBE4V23X2CNK56N5RS5A4QZ2PS7R62DNVVFE6MA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274331218 (375h33m8.200625895s) [noise=true , address=45.91.116.8:4048 , piece_id=SQM4MHUA7TV3GWFVHLKM4KMDD2NWJ2CJDYDUTORUM2KULD2EYBEQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274923944 (375h33m8.201218621s) [noise=true , address=45.91.116.14:4190 , piece_id=OY6M4TUDDBUJCJKUP6PBTG37DQTMG3E7HWBQ3AYYVWWM44QDNU5A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.274948104 (375h33m8.201242776s) [noise=true , address=45.91.116.5:4299 , piece_id=JCSR6TFDCYXD6ZV6GCXKX5XEJKFABNI5XTXY3XWA3MKXDAI7HM6A, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275645664 (375h33m8.201940343s) [noise=true , address=45.91.116.4:4937 , piece_id=6KNGXPXSYEZR2Q56VTZYNLQNIRE227F7HUKU2P5LNBJN73OFKPCA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.275743262 (375h33m8.202037953s) [noise=true , address=45.91.116.12:4227 , piece_id=MPQZGVY25SU6EMRXZYGRGXUKUL4KIYOLEL4Y3WJU7SODRYXFEMKA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.276941414 (375h33m8.203236106s) [noise=true , address=77.90.31.14:4662 , piece_id=7MGLGXOHAUGDF656E3YA5ERRIAJC37MOVLFLFPVSG6Q4KFBV3OAA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.277528968 (375h33m8.203823657s) [noise=true , address=77.90.31.18:4805 , piece_id=FHBHDNQE2H7NROVX3O3LUVFRIIJB3FJBFOBCX52TR5WM54UPI5ZQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:44:41.278121237 (375h33m8.204415924s) [noise=true , address=45.91.116.13:4155 , piece_id=NMJR6PP2WDCRXMTO5SB3NBP75YITP4N7SQC7WJCKA2KYTJIHV6EA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:05.004382596 (375h33m31.930677284s) [noise=true , address=104.36.82.7:4911 , piece_id=VDX3TS7WOSSMMBWZVWZOZI7D3C6KY3V2ALHUJ46YPGYO35C33YNQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: <nil> Jul 9 00:45:11.094109646 (375h33m38.020404335s) [noise=true , address=104.36.82.10:4078 , piece_id=SSDXQ3MFNXPUSKIDU2TK7OHOZCJ7HI7UGN7YFESNT2CVGRXRR46Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:36468->104.36.82.10:4078: use of closed network connection Jul 9 00:45:11.094166489 (375h33m38.020461179s) [noise=true , address=104.36.82.4:4804 , piece_id=IAUVRTJRNF5RXBVUSSOHX37QJX3KEQP5TSCRDNJ676EHKB34MAFQ, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:45:11.094218109 (375h33m38.020512787s) [noise=true , address=104.36.82.14:4223 , piece_id=NB3I3G56L7KKYY5M7FNLYVFHRCXK3NARKXRFJKR3QW4P7BQXHA4Q, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46810->104.36.82.14:4223: use of closed network connection Jul 9 00:45:11.094352041 (375h33m38.020646718s) [noise=true , address=104.36.82.12:4150 , piece_id=BDMXK6RCOK6LTX6FZACCH5PYHR2WRJJMPUILYJS6EMAWHFIQEXWA, seg_pos={0 0}, upload=300759]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55618->104.36.82.12:4150: use of closed network connection Jul 9 00:45:11.094400782 (375h33m38.020695476s) [seg_pos={0 0}, upload=300759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:45:11.128159115 (375h33m38.054453808s) [upload=300759]: Done uploading segments... Jul 9 00:45:11.128198296 (375h33m38.054492974s) [upload=300759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.210867642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X.......X...X................X...........XX...X.XXX...X.....] duration: 48.993131662s FAIL (2025-07-09 00:46:45 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.993396616s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)