Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03060561s FAIL (2025-07-27 00:34:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.487641602 (2h41m59.268574979s) [noise=true , address=77.90.31.13:4644 , piece_id=4L7FNU2PUPO7OYV3N6ZL5DWTL336XNDYHDYSCVEXFKKEWSJ4LL6A, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.487701637 (2h41m59.268635024s) [noise=true , address=104.36.82.55:4223 , piece_id=WD57UZBF4AKX7BFURA3ISAJZM6RUQ5IOFPCBZHF4ZA7IAHLJP3GA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.488179834 (2h41m59.269113213s) [noise=true , address=79.127.205.235:4657 , piece_id=CGOAGQQTZZHY7IY52WDKKT55ILUXKUDB7RTGLIWCG6MC4CEAJGOA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.488384882 (2h41m59.269318269s) [noise=true , address=77.90.31.15:4714 , piece_id=XXQD63XWG5KMB7TIL2PP77C4AWBNARGNEWQ3EUTGFRJC4UEW6DRA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.488882777 (2h41m59.269816172s) [noise=true , address=77.90.31.21:4902 , piece_id=HKRSE6BQMT3YIPQ62HWY3NAUPXL56TO2Y4ZQRLTB6T3RWFF7QDCQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.489467222 (2h41m59.270400614s) [noise=true , address=77.90.31.11:4579 , piece_id=EM637FGIU7ZXWCUJMTGGH5RLGORFVVGFEOYW2SKUXB7G4W6PFOFQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.490024741 (2h41m59.270958126s) [noise=true , address=104.36.82.46:4508 , piece_id=JWDLNFNJK2552QNUPJ5QHR5EZJOCJCVXY5U6WZRXNPRFNQOEEZGQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.490561936 (2h41m59.271495323s) [noise=true , address=104.36.82.47:4547 , piece_id=XIXPY7DNB42KIZT5IFF4NN5XKX6EMGMTJLVYLLXHL6QPIVJJXCJQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.491132129 (2h41m59.272065511s) [noise=true , address=156.146.43.225:4448 , piece_id=VQJQIOUKHAAGGPPKWPNB36V6YB2DTZ3CG26AVZHYURQDWWKSTWMQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.491664941 (2h41m59.272598325s) [noise=true , address=104.36.82.52:4096 , piece_id=HNI4KQC3RTYTOK6XVOJ2DYSQ7O3AGBS7B7BPNTWL4XOEYEEVTPBA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.492216720 (2h41m59.273150104s) [noise=true , address=64.49.13.138:4483 , piece_id=S573IWHSBPEMN637GQNPFVVXCDMWRBDLO2WNPVTOG4RK6BB2ISPA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.492774668 (2h41m59.273708097s) [noise=true , address=104.36.82.54:4187 , piece_id=DP7A5VUNF5DAPEPBX7KKQXZBSL7WYTOZKUEYTM3JLYHWOMLS4EOQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.493402827 (2h41m59.274336218s) [noise=true , address=79.127.201.215:4343 , piece_id=TBRQWWYUDP7SOVEEUFGTJFBNTYNP2KEJUORZHMDAZSDMDOQIWHDQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.493949554 (2h41m59.274882945s) [noise=true , address=104.36.82.2:4748 , piece_id=TG2NMK4G4DBH2ONIPCRAMLV4GQGHKLGGJ5KGKRR5Q3HRZ5Y4EGKA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.494489041 (2h41m59.27542246s) [noise=true , address=104.36.82.11:4127 , piece_id=5EV5RW3CTGV22KLLLAE7P5GIFAE76L7S6U6DCJIV6R7R6M3CGHCA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.597723882 (2h41m59.378657289s) [upload=2490]: Next returned nil segment Jul 27 00:33:43.597753328 (2h41m59.378686715s) [upload=2490]: Waiting for error group managing segments... Jul 27 00:33:43.614704527 (2h41m59.395637915s) [noise=true , address=77.90.31.11:4579 , piece_id=EM637FGIU7ZXWCUJMTGGH5RLGORFVVGFEOYW2SKUXB7G4W6PFOFQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:43.666532633 (2h41m59.447466003s) [noise=true , address=156.146.43.225:4448 , piece_id=VQJQIOUKHAAGGPPKWPNB36V6YB2DTZ3CG26AVZHYURQDWWKSTWMQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:33:44.062967841 (2h41m59.843901221s) [noise=true , address=79.127.205.234:4610 , piece_id=IGVKOIZKCF33LO4KS5EWRDJLFRAFSXNVBVDO2D6AU3OQXJVGJG6A, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:44.075951518 (2h41m59.856884908s) [noise=true , address=79.127.219.38:4001 , piece_id=MJELZ7SSO24CWDJPDHCWTW4RSLGJ6EHGKHVGLJFG55FLAQXDYXWQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:44.130316939 (2h41m59.911250308s) [noise=true , address=79.127.205.235:4657 , piece_id=CGOAGQQTZZHY7IY52WDKKT55ILUXKUDB7RTGLIWCG6MC4CEAJGOA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:45.126339405 (2h42m0.907272785s) [noise=true , address=64.49.13.138:4482 , piece_id=LAQISUWZNJQT4BB5AEVKBIS3S3CC2G7KN3FFPXSCHDSENCUHF7OQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:45.248105911 (2h42m1.029039298s) [noise=true , address=64.49.13.138:4483 , piece_id=S573IWHSBPEMN637GQNPFVVXCDMWRBDLO2WNPVTOG4RK6BB2ISPA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:33:46.857255695 (2h42m2.638189063s) [noise=true , address=77.90.31.13:4644 , piece_id=4L7FNU2PUPO7OYV3N6ZL5DWTL336XNDYHDYSCVEXFKKEWSJ4LL6A, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:34:06.788607155 (2h42m22.569540535s) [noise=true , address=79.127.201.215:4343 , piece_id=TBRQWWYUDP7SOVEEUFGTJFBNTYNP2KEJUORZHMDAZSDMDOQIWHDQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:34:43.384486319 (2h42m59.165419701s) [noise=true , address=77.90.31.21:4902 , piece_id=HKRSE6BQMT3YIPQ62HWY3NAUPXL56TO2Y4ZQRLTB6T3RWFF7QDCQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: rpc: dial tcp 77.90.31.21:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384535758 (2h42m59.165469125s) [noise=true , address=104.36.82.2:4748 , piece_id=TG2NMK4G4DBH2ONIPCRAMLV4GQGHKLGGJ5KGKRR5Q3HRZ5Y4EGKA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:104.36.82.2:4748): protocol: write tcp 159.223.143.39:43994->104.36.82.2:4748: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384561873 (2h42m59.165495248s) [noise=true , address=104.36.82.52:4096 , piece_id=HNI4KQC3RTYTOK6XVOJ2DYSQ7O3AGBS7B7BPNTWL4XOEYEEVTPBA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384630926 (2h42m59.165564288s) [noise=true , address=104.36.82.54:4187 , piece_id=DP7A5VUNF5DAPEPBX7KKQXZBSL7WYTOZKUEYTM3JLYHWOMLS4EOQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: write tcp 159.223.143.39:34234->104.36.82.54:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384638968 (2h42m59.165572332s) [noise=true , address=77.90.31.15:4714 , piece_id=XXQD63XWG5KMB7TIL2PP77C4AWBNARGNEWQ3EUTGFRJC4UEW6DRA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384720236 (2h42m59.165653614s) [noise=true , address=104.36.82.46:4508 , piece_id=JWDLNFNJK2552QNUPJ5QHR5EZJOCJCVXY5U6WZRXNPRFNQOEEZGQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384728709 (2h42m59.165662074s) [noise=true , address=104.36.82.47:4547 , piece_id=XIXPY7DNB42KIZT5IFF4NN5XKX6EMGMTJLVYLLXHL6QPIVJJXCJQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: rpc: dial tcp 104.36.82.47:4547: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384785454 (2h42m59.165718819s) [noise=true , address=104.36.82.55:4223 , piece_id=WD57UZBF4AKX7BFURA3ISAJZM6RUQ5IOFPCBZHF4ZA7IAHLJP3GA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: rpc: dial tcp 104.36.82.55:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384803466 (2h42m59.165736852s) [noise=true , address=104.36.82.11:4127 , piece_id=5EV5RW3CTGV22KLLLAE7P5GIFAE76L7S6U6DCJIV6R7R6M3CGHCA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: write tcp 159.223.143.39:47242->104.36.82.11:4127: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384845238 (2h42m59.165778624s) [seg_pos={0 0}, upload=2490]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:34:43.411651581 (2h42m59.192584981s) [upload=2490]: Done uploading segments... Jul 27 00:34:43.411684123 (2h42m59.192617516s) [upload=2490]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.030633949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X........X...XX...............................X.............] duration: 38.311359952s FAIL (2025-07-27 00:34:37 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.311387923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X......X.XX....X...X.................XX....X......XX.XXX] duration: 30.000667811s FAIL (2025-07-27 00:34:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000685187s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.] duration: 1.878007842s FAIL (2025-07-27 00:34:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nnection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780254899 (2h41m44.561188289s) [seg_pos={0 0}, upload=2481]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:33:28.813631340 (2h41m44.594564717s) [upload=2481]: Done uploading segments... Jul 27 00:33:28.813655080 (2h41m44.594588456s) [upload=2481]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:33:42.772431891 (2h41m58.55336527s) [upload=2489]: Starting upload Jul 27 00:33:42.772531982 (2h41m58.553465363s) [upload=2489]: Done starting upload Jul 27 00:33:42.772598210 (2h41m58.553531573s) [upload=2489]: Uploading segments... Jul 27 00:33:42.772897554 (2h41m58.553830942s) [upload=2489]: Got next segment. Inline: false Jul 27 00:33:42.772966928 (2h41m58.55390031s) [upload=2489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:33:42.772973908 (2h41m58.553907289s) [upload=2489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:33:42.809452437 (2h41m58.590385807s) [seg_pos={0 0}, upload=2489]: Begin upload segment... Jul 27 00:33:42.809511211 (2h41m58.590444574s) [seg_pos={0 0}, upload=2489]: Done begin upload segment. Jul 27 00:33:42.809548227 (2h41m58.590481613s) [noise=true , address=104.36.82.14:4223 , piece_id=S5EWQ2GVNMNPUM2QA72JM3Y55KVRLPX7XIGKMXJMTB2HUHHBAY4A, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.810515534 (2h41m58.591448926s) [noise=true , address=45.91.116.10:4083 , piece_id=2A3PCZSWH6MTCNGZSR3ZEMQPYLQ6S2O4DHBBOPKELCIVVTMFXGDA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.810908555 (2h41m58.591841935s) [noise=true , address=45.91.116.13:4154 , piece_id=PHANYSQICLI44COK56IY5MMOQKSZQBG2J3OOJV73CVD66SD3BFWA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.813631673 (2h41m58.594565056s) [noise=true , address=104.36.82.10:4078 , piece_id=Q3UJWKRHCJEFIAHX7PTAHCKID3OSZIN5JAUOCOJGIRZ7AGQMHWEA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.814816475 (2h41m58.595749864s) [noise=true , address=45.91.116.4:4940 , piece_id=YUG6VMXYZDGH3IABFW4XF43TU4ZQGSNITXDOCIIDUYQ4ZZXIYC7A, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.815421088 (2h41m58.596354466s) [noise=true , address=104.36.82.4:4804 , piece_id=KZBEMTFRGP7LJKTCF5ZVFBJH5EFFGQIMNTICJ7U3FEHOPRALWKXQ, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816024223 (2h41m58.59695761s) [noise=true , address=45.91.116.12:4226 , piece_id=FMD6MUY7UVHCBIYFD2BJX7WKJOGKCQUX56OT6ENPTDTYUUC2BYJA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816505852 (2h41m58.597439232s) [noise=true , address=77.90.31.16:4735 , piece_id=X4EK2C2UXB74VDQHCUYMVZ2TGHWBT4XIKHZH53XEANZLYNDIZ7JA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816981623 (2h41m58.597915006s) [noise=true , address=77.90.31.14:4662 , piece_id=UQFCPUYMOPYTHB7JMKJMD4XHN6GAQNJTA6DNJEQBPEGKWBSIJEFQ, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.817406255 (2h41m58.598339641s) [noise=true , address=45.91.116.11:4119 , piece_id=UX4UFKU5UFJVBF575IAWUJDPTUDANNDCAZA5BZLTGDKGEJK3TD7Q, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.817912249 (2h41m58.598845634s) [noise=true , address=45.91.116.6:4973 , piece_id=UZDWLZZVEB7G7CQXLQDFQBWLDFEPJP47FSQXQT4QI6RCWHOX37PA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.818405766 (2h41m58.599339154s) [noise=true , address=104.36.82.13:4187 , piece_id=VOUFP55K63GOPO7352M43MOF6OOZAMGGNNFWOZBPNTIYMYIP47XA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.818892203 (2h41m58.599825593s) [noise=true , address=45.91.116.14:4191 , piece_id=UVT23XLMFU7O6JVGPNGLGEYJBA27LCGESLADL3SOQNRFBZYFAPNA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.819335313 (2h41m58.600268697s) [noise=true , address=45.91.116.6:4974 , piece_id=QKXO7XNRATDMWCNIVSP6E6KWD4QBXK56R4HEPCEWAE6TIS4GLFQA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.819881361 (2h41m58.60081475s) [noise=true , address=104.36.82.3:4767 , piece_id=O2WDAQJY4RBQVZ72EVWVBYZX44F7TLDMTL6GYCN3XG6VF2ILA67Q, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.820392722 (2h41m58.601326112s) [noise=true , address=104.36.82.9:4007 , piece_id=7D6VCXWVHACS3MKG2VT6RFPPDLKSMWQEZEDOS6FJ4A44YXULY5NA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.971323604 (2h41m58.752256991s) [upload=2489]: Next returned nil segment Jul 27 00:33:42.971346037 (2h41m58.75227942s) [upload=2489]: Waiting for error group managing segments... Jul 27 00:33:42.976454805 (2h41m58.757388175s) [noise=true , address=45.91.116.6:4974 , piece_id=QKXO7XNRATDMWCNIVSP6E6KWD4QBXK56R4HEPCEWAE6TIS4GLFQA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.979574580 (2h41m58.76050795s) [noise=true , address=45.91.116.10:4083 , piece_id=2A3PCZSWH6MTCNGZSR3ZEMQPYLQ6S2O4DHBBOPKELCIVVTMFXGDA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.979620139 (2h41m58.760553522s) [noise=true , address=45.91.116.12:4226 , piece_id=FMD6MUY7UVHCBIYFD2BJX7WKJOGKCQUX56OT6ENPTDTYUUC2BYJA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.980316822 (2h41m58.761250191s) [noise=true , address=45.91.116.6:4973 , piece_id=UZDWLZZVEB7G7CQXLQDFQBWLDFEPJP47FSQXQT4QI6RCWHOX37PA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.980396243 (2h41m58.761329628s) [noise=true , address=45.91.116.13:4154 , piece_id=PHANYSQICLI44COK56IY5MMOQKSZQBG2J3OOJV73CVD66SD3BFWA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.981311003 (2h41m58.762244372s) [noise=true , address=45.91.116.11:4119 , piece_id=UX4UFKU5UFJVBF575IAWUJDPTUDANNDCAZA5BZLTGDKGEJK3TD7Q, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.981996571 (2h41m58.762929941s) [noise=true , address=45.91.116.4:4940 , piece_id=YUG6VMXYZDGH3IABFW4XF43TU4ZQGSNITXDOCIIDUYQ4ZZXIYC7A, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.982071090 (2h41m58.763004455s) [noise=true , address=77.90.31.16:4735 , piece_id=X4EK2C2UXB74VDQHCUYMVZ2TGHWBT4XIKHZH53XEANZLYNDIZ7JA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.982578044 (2h41m58.763511411s) [noise=true , address=77.90.31.14:4662 , piece_id=UQFCPUYMOPYTHB7JMKJMD4XHN6GAQNJTA6DNJEQBPEGKWBSIJEFQ, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:43.139592182 (2h41m58.920525563s) [noise=true , address=45.91.116.14:4191 , piece_id=UVT23XLMFU7O6JVGPNGLGEYJBA27LCGESLADL3SOQNRFBZYFAPNA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:34:12.742401054 (2h42m28.523334439s) [noise=true , address=104.36.82.3:4767 , piece_id=O2WDAQJY4RBQVZ72EVWVBYZX44F7TLDMTL6GYCN3XG6VF2ILA67Q, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742488371 (2h42m28.523421747s) [noise=true , address=104.36.82.9:4007 , piece_id=7D6VCXWVHACS3MKG2VT6RFPPDLKSMWQEZEDOS6FJ4A44YXULY5NA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742542194 (2h42m28.523475563s) [noise=true , address=104.36.82.13:4187 , piece_id=VOUFP55K63GOPO7352M43MOF6OOZAMGGNNFWOZBPNTIYMYIP47XA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:43136->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742614945 (2h42m28.523548309s) [noise=true , address=104.36.82.14:4223 , piece_id=S5EWQ2GVNMNPUM2QA72JM3Y55KVRLPX7XIGKMXJMTB2HUHHBAY4A, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:41060->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742668062 (2h42m28.523601432s) [noise=true , address=104.36.82.4:4804 , piece_id=KZBEMTFRGP7LJKTCF5ZVFBJH5EFFGQIMNTICJ7U3FEHOPRALWKXQ, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:46106->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742752541 (2h42m28.523685906s) [noise=true , address=104.36.82.10:4078 , piece_id=Q3UJWKRHCJEFIAHX7PTAHCKID3OSZIN5JAUOCOJGIRZ7AGQMHWEA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:59466->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742794169 (2h42m28.523727553s) [seg_pos={0 0}, upload=2489]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:34:12.770005947 (2h42m28.550939315s) [upload=2489]: Done uploading segments... Jul 27 00:34:12.770043270 (2h42m28.550976637s) [upload=2489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.878035591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX] duration: 30.029444024s FAIL (2025-07-27 00:34:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nnection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780254899 (2h41m44.561188289s) [seg_pos={0 0}, upload=2481]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:33:28.813631340 (2h41m44.594564717s) [upload=2481]: Done uploading segments... Jul 27 00:33:28.813655080 (2h41m44.594588456s) [upload=2481]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:33:42.772431891 (2h41m58.55336527s) [upload=2489]: Starting upload Jul 27 00:33:42.772531982 (2h41m58.553465363s) [upload=2489]: Done starting upload Jul 27 00:33:42.772598210 (2h41m58.553531573s) [upload=2489]: Uploading segments... Jul 27 00:33:42.772897554 (2h41m58.553830942s) [upload=2489]: Got next segment. Inline: false Jul 27 00:33:42.772966928 (2h41m58.55390031s) [upload=2489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:33:42.772973908 (2h41m58.553907289s) [upload=2489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:33:42.809452437 (2h41m58.590385807s) [seg_pos={0 0}, upload=2489]: Begin upload segment... Jul 27 00:33:42.809511211 (2h41m58.590444574s) [seg_pos={0 0}, upload=2489]: Done begin upload segment. Jul 27 00:33:42.809548227 (2h41m58.590481613s) [noise=true , address=104.36.82.14:4223 , piece_id=S5EWQ2GVNMNPUM2QA72JM3Y55KVRLPX7XIGKMXJMTB2HUHHBAY4A, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.810515534 (2h41m58.591448926s) [noise=true , address=45.91.116.10:4083 , piece_id=2A3PCZSWH6MTCNGZSR3ZEMQPYLQ6S2O4DHBBOPKELCIVVTMFXGDA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.810908555 (2h41m58.591841935s) [noise=true , address=45.91.116.13:4154 , piece_id=PHANYSQICLI44COK56IY5MMOQKSZQBG2J3OOJV73CVD66SD3BFWA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.813631673 (2h41m58.594565056s) [noise=true , address=104.36.82.10:4078 , piece_id=Q3UJWKRHCJEFIAHX7PTAHCKID3OSZIN5JAUOCOJGIRZ7AGQMHWEA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.814816475 (2h41m58.595749864s) [noise=true , address=45.91.116.4:4940 , piece_id=YUG6VMXYZDGH3IABFW4XF43TU4ZQGSNITXDOCIIDUYQ4ZZXIYC7A, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.815421088 (2h41m58.596354466s) [noise=true , address=104.36.82.4:4804 , piece_id=KZBEMTFRGP7LJKTCF5ZVFBJH5EFFGQIMNTICJ7U3FEHOPRALWKXQ, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816024223 (2h41m58.59695761s) [noise=true , address=45.91.116.12:4226 , piece_id=FMD6MUY7UVHCBIYFD2BJX7WKJOGKCQUX56OT6ENPTDTYUUC2BYJA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816505852 (2h41m58.597439232s) [noise=true , address=77.90.31.16:4735 , piece_id=X4EK2C2UXB74VDQHCUYMVZ2TGHWBT4XIKHZH53XEANZLYNDIZ7JA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816981623 (2h41m58.597915006s) [noise=true , address=77.90.31.14:4662 , piece_id=UQFCPUYMOPYTHB7JMKJMD4XHN6GAQNJTA6DNJEQBPEGKWBSIJEFQ, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.817406255 (2h41m58.598339641s) [noise=true , address=45.91.116.11:4119 , piece_id=UX4UFKU5UFJVBF575IAWUJDPTUDANNDCAZA5BZLTGDKGEJK3TD7Q, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.817912249 (2h41m58.598845634s) [noise=true , address=45.91.116.6:4973 , piece_id=UZDWLZZVEB7G7CQXLQDFQBWLDFEPJP47FSQXQT4QI6RCWHOX37PA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.818405766 (2h41m58.599339154s) [noise=true , address=104.36.82.13:4187 , piece_id=VOUFP55K63GOPO7352M43MOF6OOZAMGGNNFWOZBPNTIYMYIP47XA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.818892203 (2h41m58.599825593s) [noise=true , address=45.91.116.14:4191 , piece_id=UVT23XLMFU7O6JVGPNGLGEYJBA27LCGESLADL3SOQNRFBZYFAPNA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.819335313 (2h41m58.600268697s) [noise=true , address=45.91.116.6:4974 , piece_id=QKXO7XNRATDMWCNIVSP6E6KWD4QBXK56R4HEPCEWAE6TIS4GLFQA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.819881361 (2h41m58.60081475s) [noise=true , address=104.36.82.3:4767 , piece_id=O2WDAQJY4RBQVZ72EVWVBYZX44F7TLDMTL6GYCN3XG6VF2ILA67Q, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.820392722 (2h41m58.601326112s) [noise=true , address=104.36.82.9:4007 , piece_id=7D6VCXWVHACS3MKG2VT6RFPPDLKSMWQEZEDOS6FJ4A44YXULY5NA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.971323604 (2h41m58.752256991s) [upload=2489]: Next returned nil segment Jul 27 00:33:42.971346037 (2h41m58.75227942s) [upload=2489]: Waiting for error group managing segments... Jul 27 00:33:42.976454805 (2h41m58.757388175s) [noise=true , address=45.91.116.6:4974 , piece_id=QKXO7XNRATDMWCNIVSP6E6KWD4QBXK56R4HEPCEWAE6TIS4GLFQA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.979574580 (2h41m58.76050795s) [noise=true , address=45.91.116.10:4083 , piece_id=2A3PCZSWH6MTCNGZSR3ZEMQPYLQ6S2O4DHBBOPKELCIVVTMFXGDA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.979620139 (2h41m58.760553522s) [noise=true , address=45.91.116.12:4226 , piece_id=FMD6MUY7UVHCBIYFD2BJX7WKJOGKCQUX56OT6ENPTDTYUUC2BYJA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.980316822 (2h41m58.761250191s) [noise=true , address=45.91.116.6:4973 , piece_id=UZDWLZZVEB7G7CQXLQDFQBWLDFEPJP47FSQXQT4QI6RCWHOX37PA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.980396243 (2h41m58.761329628s) [noise=true , address=45.91.116.13:4154 , piece_id=PHANYSQICLI44COK56IY5MMOQKSZQBG2J3OOJV73CVD66SD3BFWA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.981311003 (2h41m58.762244372s) [noise=true , address=45.91.116.11:4119 , piece_id=UX4UFKU5UFJVBF575IAWUJDPTUDANNDCAZA5BZLTGDKGEJK3TD7Q, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.981996571 (2h41m58.762929941s) [noise=true , address=45.91.116.4:4940 , piece_id=YUG6VMXYZDGH3IABFW4XF43TU4ZQGSNITXDOCIIDUYQ4ZZXIYC7A, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.982071090 (2h41m58.763004455s) [noise=true , address=77.90.31.16:4735 , piece_id=X4EK2C2UXB74VDQHCUYMVZ2TGHWBT4XIKHZH53XEANZLYNDIZ7JA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.982578044 (2h41m58.763511411s) [noise=true , address=77.90.31.14:4662 , piece_id=UQFCPUYMOPYTHB7JMKJMD4XHN6GAQNJTA6DNJEQBPEGKWBSIJEFQ, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:43.139592182 (2h41m58.920525563s) [noise=true , address=45.91.116.14:4191 , piece_id=UVT23XLMFU7O6JVGPNGLGEYJBA27LCGESLADL3SOQNRFBZYFAPNA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:34:12.742401054 (2h42m28.523334439s) [noise=true , address=104.36.82.3:4767 , piece_id=O2WDAQJY4RBQVZ72EVWVBYZX44F7TLDMTL6GYCN3XG6VF2ILA67Q, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742488371 (2h42m28.523421747s) [noise=true , address=104.36.82.9:4007 , piece_id=7D6VCXWVHACS3MKG2VT6RFPPDLKSMWQEZEDOS6FJ4A44YXULY5NA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742542194 (2h42m28.523475563s) [noise=true , address=104.36.82.13:4187 , piece_id=VOUFP55K63GOPO7352M43MOF6OOZAMGGNNFWOZBPNTIYMYIP47XA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:43136->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742614945 (2h42m28.523548309s) [noise=true , address=104.36.82.14:4223 , piece_id=S5EWQ2GVNMNPUM2QA72JM3Y55KVRLPX7XIGKMXJMTB2HUHHBAY4A, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:41060->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742668062 (2h42m28.523601432s) [noise=true , address=104.36.82.4:4804 , piece_id=KZBEMTFRGP7LJKTCF5ZVFBJH5EFFGQIMNTICJ7U3FEHOPRALWKXQ, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:46106->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742752541 (2h42m28.523685906s) [noise=true , address=104.36.82.10:4078 , piece_id=Q3UJWKRHCJEFIAHX7PTAHCKID3OSZIN5JAUOCOJGIRZ7AGQMHWEA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:59466->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742794169 (2h42m28.523727553s) [seg_pos={0 0}, upload=2489]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:34:12.770005947 (2h42m28.550939315s) [upload=2489]: Done uploading segments... Jul 27 00:34:12.770043270 (2h42m28.550976637s) [upload=2489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.029509443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.X] duration: 30.029750509s FAIL (2025-07-27 00:34:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.029781575s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 230.506961ms FAIL (2025-07-27 00:34:01 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 230.559805ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X..X......X.XX....X...X.................XX....X......XX.XX] duration: 30.0017946s FAIL (2025-07-27 00:33:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001816316s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033159718s FAIL (2025-07-27 00:33:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== WR73Q5YM47RAPWB7U4VHUUGJM7GRB7ESRIJOICZQLUXQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.873653013 (2h40m48.654586394s) [noise=true , address=45.91.116.10:4093 , piece_id=63H2L4LKTHH5QYRV4TOP6ZQONA323DTCVWGPZBPLPIF6XV4TQ5YQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.874037487 (2h40m48.654970871s) [noise=true , address=79.127.201.214:4289 , piece_id=ZZK7PTB6CDICHZDG72FX57UJCXM5ICHBKQ5GOYE65HOEOBN5LKVA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.874941234 (2h40m48.655874598s) [noise=true , address=104.36.82.51:4080 , piece_id=SPAJBTBULQT3RVPY6FUDQHK64I25GPWF3R5S5TTHF72DV5XVPXJA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.875346425 (2h40m48.656279801s) [noise=true , address=104.36.82.52:4093 , piece_id=PWFRI4F2PJV4TZM4YCZD2VG6CU7E7NEAYFOBQVVPEF7MTVF7Y4TA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.875723827 (2h40m48.656657212s) [noise=true , address=77.90.31.13:4631 , piece_id=2VRJ7AUGFDKW47E5JXAAUSU6OU7GBX2HPT43D3QQN232UHUFZKYQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.876233741 (2h40m48.657167196s) [noise=true , address=104.36.82.56:4245 , piece_id=YDIU2HDIEVCJIUB7NR33FJOQ4FRKQLDONMX4CTIRBOC2VK3HS5CQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.876656791 (2h40m48.657590177s) [noise=true , address=45.91.116.4:4964 , piece_id=3V6OQAYV3L5JPKDL6H3FRFDWXQOD4ZI5FE7PAOGSN3WFTVYBTQLA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.877721857 (2h40m48.658655238s) [noise=true , address=79.127.201.215:4357 , piece_id=NDW4CGTHNJF7AUE4ORYMYFSYTIS65IUFBV6B7ZANCGDHH66U5VBA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.878154730 (2h40m48.659088116s) [noise=true , address=64.49.13.140:4514 , piece_id=EAHKRZNAI4ZYTYG3DUDO563HPFOEOAOLTVC2TZJO23H3Q2YLWSTQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.878549266 (2h40m48.659482652s) [noise=true , address=79.127.205.234:4613 , piece_id=JNPUMNDJ3TOUURX4RJ4UUUH5X4Q4P5V46YRR3PPBY2UVKIIWGP5Q, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.878887423 (2h40m48.659820809s) [noise=true , address=79.127.201.214:4321 , piece_id=QGPKUL7XHWNN36HZVCCKTRUOZ6IOTHRUEK3TT7TP4R6Y6KFORUWA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.881242445 (2h40m48.662175831s) [noise=true , address=156.146.43.225:4448 , piece_id=QSQWT6TYZI46MRXIXYN67BBVZBB257NGYWKDG4Y2SH4QXZSAMVHA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.881751508 (2h40m48.662684892s) [noise=true , address=79.127.219.38:4007 , piece_id=5GNBYBDZSXHP4DGTGPEKXOFMAMYGWEQZJ246KTQPU6VNXVPS5JEQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.882091662 (2h40m48.663025055s) [noise=true , address=104.36.82.46:4488 , piece_id=MRJLUZQCZ6QBOEPZKXGXY3T7E7WG35HIZAWABFATEHD4XV2JIW2Q, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.882409721 (2h40m48.663343105s) [noise=true , address=79.127.205.245:4244 , piece_id=TZ2V2R34IL7D6LBYNVA4YCRIP4FTT6ASLJQ527OH7UIBFPBDFXWA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.882707035 (2h40m48.663640415s) [noise=true , address=104.36.82.47:4517 , piece_id=3UTUJZV2EF5DPTJYUWHJIRJO52L5FTAJK6AVFSR5B5OBB7VHXKJQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:33.003096411 (2h40m48.784029779s) [upload=2473]: Next returned nil segment Jul 27 00:32:33.003120023 (2h40m48.784053406s) [upload=2473]: Waiting for error group managing segments... Jul 27 00:32:33.045919038 (2h40m48.826852416s) [noise=true , address=79.127.219.38:4007 , piece_id=5GNBYBDZSXHP4DGTGPEKXOFMAMYGWEQZJ246KTQPU6VNXVPS5JEQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:33.061201994 (2h40m48.842135374s) [noise=true , address=79.127.201.214:4289 , piece_id=ZZK7PTB6CDICHZDG72FX57UJCXM5ICHBKQ5GOYE65HOEOBN5LKVA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1YfeEiw7zKAYgUcMqMLgEiS2t1gMqdwrpuAdTb2dZouuXRBSM4, address:79.127.201.214:4289): protocol: storage node overloaded, request limit: 2000 Jul 27 00:32:33.500067952 (2h40m49.281001317s) [noise=true , address=79.127.205.245:4244 , piece_id=TZ2V2R34IL7D6LBYNVA4YCRIP4FTT6ASLJQ527OH7UIBFPBDFXWA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:33.509689087 (2h40m49.290622452s) [noise=true , address=79.127.205.246:4280 , piece_id=2K5GGHS6WR73Q5YM47RAPWB7U4VHUUGJM7GRB7ESRIJOICZQLUXQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:33.534479827 (2h40m49.31541319s) [noise=true , address=79.127.205.234:4613 , piece_id=JNPUMNDJ3TOUURX4RJ4UUUH5X4Q4P5V46YRR3PPBY2UVKIIWGP5Q, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:35.162221330 (2h40m50.943154699s) [noise=true , address=156.146.43.225:4448 , piece_id=QSQWT6TYZI46MRXIXYN67BBVZBB257NGYWKDG4Y2SH4QXZSAMVHA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:35.330783593 (2h40m51.111716954s) [noise=true , address=79.127.201.214:4321 , piece_id=QGPKUL7XHWNN36HZVCCKTRUOZ6IOTHRUEK3TT7TP4R6Y6KFORUWA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:35.392901462 (2h40m51.173834829s) [noise=true , address=64.49.13.140:4514 , piece_id=EAHKRZNAI4ZYTYG3DUDO563HPFOEOAOLTVC2TZJO23H3Q2YLWSTQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 27 00:32:35.620578098 (2h40m51.401511463s) [noise=true , address=45.91.116.4:4964 , piece_id=3V6OQAYV3L5JPKDL6H3FRFDWXQOD4ZI5FE7PAOGSN3WFTVYBTQLA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:45.927439038 (2h41m1.708372433s) [noise=true , address=45.91.116.10:4093 , piece_id=63H2L4LKTHH5QYRV4TOP6ZQONA323DTCVWGPZBPLPIF6XV4TQ5YQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:32:46.666851704 (2h41m2.447785069s) [noise=true , address=79.127.201.215:4357 , piece_id=NDW4CGTHNJF7AUE4ORYMYFSYTIS65IUFBV6B7ZANCGDHH66U5VBA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:33:32.790751529 (2h41m48.571684909s) [noise=true , address=77.90.31.13:4631 , piece_id=2VRJ7AUGFDKW47E5JXAAUSU6OU7GBX2HPT43D3QQN232UHUFZKYQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:33:32.790845734 (2h41m48.571779114s) [noise=true , address=104.36.82.46:4488 , piece_id=MRJLUZQCZ6QBOEPZKXGXY3T7E7WG35HIZAWABFATEHD4XV2JIW2Q, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: write tcp 159.223.143.39:41762->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791293046 (2h41m48.572226424s) [noise=true , address=104.36.82.52:4093 , piece_id=PWFRI4F2PJV4TZM4YCZD2VG6CU7E7NEAYFOBQVVPEF7MTVF7Y4TA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: write tcp 159.223.143.39:35142->104.36.82.52:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791347335 (2h41m48.572280715s) [noise=true , address=104.36.82.47:4517 , piece_id=3UTUJZV2EF5DPTJYUWHJIRJO52L5FTAJK6AVFSR5B5OBB7VHXKJQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: write tcp 159.223.143.39:48942->104.36.82.47:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791392426 (2h41m48.572325804s) [noise=true , address=104.36.82.51:4080 , piece_id=SPAJBTBULQT3RVPY6FUDQHK64I25GPWF3R5S5TTHF72DV5XVPXJA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C, address:104.36.82.51:4080): protocol: write tcp 159.223.143.39:44202->104.36.82.51:4080: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791536688 (2h41m48.572470067s) [noise=true , address=104.36.82.56:4245 , piece_id=YDIU2HDIEVCJIUB7NR33FJOQ4FRKQLDONMX4CTIRBOC2VK3HS5CQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12YnxyH9TZVuVheQpH9NVRfFUeWHJEtV97wyEJafhCzGZSv13Fn, address:104.36.82.56:4245): protocol: write tcp 159.223.143.39:50252->104.36.82.56:4245: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791749070 (2h41m48.572682448s) [noise=true , address=104.36.82.54:4192 , piece_id=3WY4POFOJ4A62SFUP3D5C2PEGT6Z7LFLEPKD4KCKSL25BN4LU2HA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: write tcp 159.223.143.39:58184->104.36.82.54:4192: use of closed network connection; context deadline exceeded Jul 27 00:33:32.791943618 (2h41m48.572877012s) [seg_pos={0 0}, upload=2473]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:33:32.818918383 (2h41m48.599851767s) [upload=2473]: Done uploading segments... Jul 27 00:33:32.818965558 (2h41m48.599898942s) [upload=2473]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.033184486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X........X...XX...............................X............] duration: 47.041645583s FAIL (2025-07-27 00:33:31 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.041699455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.X] duration: 30.036525072s FAIL (2025-07-27 00:33:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== osed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:14.589446796 (2h40m30.370380174s) [seg_pos={0 0}, upload=2466]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:32:14.589537039 (2h40m30.370470418s) [upload=2466]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:32:14.589542609 (2h40m30.370475994s) [upload=2466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:32:14.640249492 (2h40m30.421182861s) [upload=2466]: Done uploading segments... Jul 27 00:32:14.640270518 (2h40m30.421203886s) [upload=2466]: Upload finished. err: <nil> Jul 27 00:32:58.810974159 (2h41m14.591907543s) [upload=2481]: Starting upload Jul 27 00:32:58.811109781 (2h41m14.592043169s) [upload=2481]: Done starting upload Jul 27 00:32:58.811265564 (2h41m14.592198942s) [upload=2481]: Uploading segments... Jul 27 00:32:58.811695042 (2h41m14.592628422s) [upload=2481]: Got next segment. Inline: false Jul 27 00:32:58.811886987 (2h41m14.592820376s) [upload=2481]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:32:58.811897306 (2h41m14.592830689s) [upload=2481]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:32:58.860666254 (2h41m14.641599642s) [seg_pos={0 0}, upload=2481]: Begin upload segment... Jul 27 00:32:58.860753313 (2h41m14.641686696s) [seg_pos={0 0}, upload=2481]: Done begin upload segment. Jul 27 00:32:58.860816149 (2h41m14.641749548s) [noise=true , address=77.90.31.16:4734 , piece_id=Y6YU54NGQFFKDDYM3NLHHACBJU3O5POX6QFLOQK56IRSPQ6BO4SA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.861286105 (2h41m14.642219492s) [noise=true , address=45.91.116.13:4153 , piece_id=F5DFROAFAGVTGEVWIIIYCM65AQIG5F5L6DJ27D2VJWBKGJP3HDKA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.861698316 (2h41m14.6426317s) [noise=true , address=77.90.31.14:4663 , piece_id=IXAJPTO6ZLN5DPISKO6LOQ7FATM4LZV4MZETLMPMJS7MP62VO5TA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.862003887 (2h41m14.642937267s) [noise=true , address=104.36.82.7:4912 , piece_id=MQQPZESGTLSRHZ57V4EYB3EZOILZHAF5HLZM67GXFIXIHUVSCJWQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.862312037 (2h41m14.643245454s) [noise=true , address=77.90.31.13:4628 , piece_id=MIOP42HZ77XLN4Z7FIVKQVRAXGHLB5QJUHSLNTLLY6LK4IYR3BVQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.862721254 (2h41m14.643654639s) [noise=true , address=77.90.31.15:4697 , piece_id=VOQWGYE32QQPU3RYAPN36N7ED4XM7JEIF6IFP7OOIB6YKV3IAVLA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.863184293 (2h41m14.64411768s) [noise=true , address=77.90.31.19:4841 , piece_id=G4MUHCH3OG4YFYS722XM42ZUUIST6YGEWQGVQL26GXYCOJVCDLBA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.863205444 (2h41m14.644138827s) [noise=true , address=104.36.82.5:4838 , piece_id=TII2TRXDFVNXUVYHNJGQBYVNJRKZFP5AYS26A74KN4HUUHLD63KQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.863713171 (2h41m14.644646559s) [noise=true , address=45.91.116.5:4297 , piece_id=EEO45A4W7UN7MNXUOT3PGZTQ5XYKWCI3Y4JNHFH2CAWEMGVHMUQQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.866569418 (2h41m14.647502805s) [noise=true , address=104.36.82.14:4221 , piece_id=7UH4FTXZYU7DGK2KDU6QKZT3JZQHJBB6BS6CDVFPYGU6KUHLKGWA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.866984782 (2h41m14.647918159s) [noise=true , address=45.91.116.4:4939 , piece_id=RVEGNB5FPYKPMPXTZ7CZTKAFLKHWLSG6JQRI3D6EP2UTBAFPPSNQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.868444601 (2h41m14.649377988s) [noise=true , address=104.36.82.6:4873 , piece_id=KT3PFAPXSI2R7VKIOWG4BLQRLFYGHOYZWNVKJ2WFLHRUW74UO5ZA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.868895493 (2h41m14.64982888s) [noise=true , address=45.91.116.7:4011 , piece_id=X7QDYAS43OY7H47XQ7APQ2BD6D2JE2IAFDIRDMNR377E5GTAWOSQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.869368113 (2h41m14.650301501s) [noise=true , address=45.91.116.8:4048 , piece_id=LVODTJLTSPJRFRA76P2WFKCPQFXTYSXZOUCRVK3NJLQNYYDUDFMA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.869959041 (2h41m14.650892443s) [noise=true , address=45.91.116.11:4117 , piece_id=BTDJPKB62ETXNI5VGCJH7NAVSJPLHC5ATIU6VTEMQARXFHPODP7Q, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.870369000 (2h41m14.651302363s) [noise=true , address=45.91.116.8:4047 , piece_id=G5W7W2CY5ZPRFROVZWQQO76JE4276FG44FMH7YYYSISEHH2OPF7A, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.973013076 (2h41m14.753946461s) [upload=2481]: Next returned nil segment Jul 27 00:32:58.973035721 (2h41m14.753969117s) [upload=2481]: Waiting for error group managing segments... Jul 27 00:32:58.977494008 (2h41m14.758427389s) [noise=true , address=45.91.116.4:4939 , piece_id=RVEGNB5FPYKPMPXTZ7CZTKAFLKHWLSG6JQRI3D6EP2UTBAFPPSNQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.978167121 (2h41m14.759100501s) [noise=true , address=45.91.116.11:4117 , piece_id=BTDJPKB62ETXNI5VGCJH7NAVSJPLHC5ATIU6VTEMQARXFHPODP7Q, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.978724285 (2h41m14.759657669s) [noise=true , address=45.91.116.7:4011 , piece_id=X7QDYAS43OY7H47XQ7APQ2BD6D2JE2IAFDIRDMNR377E5GTAWOSQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.979432936 (2h41m14.760366316s) [noise=true , address=45.91.116.8:4048 , piece_id=LVODTJLTSPJRFRA76P2WFKCPQFXTYSXZOUCRVK3NJLQNYYDUDFMA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.981203510 (2h41m14.762136873s) [noise=true , address=77.90.31.14:4663 , piece_id=IXAJPTO6ZLN5DPISKO6LOQ7FATM4LZV4MZETLMPMJS7MP62VO5TA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.983091363 (2h41m14.764024739s) [noise=true , address=45.91.116.8:4047 , piece_id=G5W7W2CY5ZPRFROVZWQQO76JE4276FG44FMH7YYYSISEHH2OPF7A, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.983281795 (2h41m14.764215174s) [noise=true , address=45.91.116.13:4153 , piece_id=F5DFROAFAGVTGEVWIIIYCM65AQIG5F5L6DJ27D2VJWBKGJP3HDKA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.984095977 (2h41m14.765029343s) [noise=true , address=45.91.116.5:4297 , piece_id=EEO45A4W7UN7MNXUOT3PGZTQ5XYKWCI3Y4JNHFH2CAWEMGVHMUQQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:59.006215392 (2h41m14.787148773s) [noise=true , address=77.90.31.16:4734 , piece_id=Y6YU54NGQFFKDDYM3NLHHACBJU3O5POX6QFLOQK56IRSPQ6BO4SA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:33:00.525382321 (2h41m16.306315703s) [noise=true , address=77.90.31.19:4841 , piece_id=G4MUHCH3OG4YFYS722XM42ZUUIST6YGEWQGVQL26GXYCOJVCDLBA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:33:01.014116504 (2h41m16.795049889s) [noise=true , address=77.90.31.13:4628 , piece_id=MIOP42HZ77XLN4Z7FIVKQVRAXGHLB5QJUHSLNTLLY6LK4IYR3BVQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:33:28.779965871 (2h41m44.56089926s) [noise=true , address=104.36.82.5:4838 , piece_id=TII2TRXDFVNXUVYHNJGQBYVNJRKZFP5AYS26A74KN4HUUHLD63KQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:35344->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780050817 (2h41m44.560984198s) [noise=true , address=104.36.82.14:4221 , piece_id=7UH4FTXZYU7DGK2KDU6QKZT3JZQHJBB6BS6CDVFPYGU6KUHLKGWA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:43084->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780117223 (2h41m44.561050604s) [noise=true , address=104.36.82.7:4912 , piece_id=MQQPZESGTLSRHZ57V4EYB3EZOILZHAF5HLZM67GXFIXIHUVSCJWQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:42662->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780175681 (2h41m44.56110907s) [noise=true , address=77.90.31.15:4697 , piece_id=VOQWGYE32QQPU3RYAPN36N7ED4XM7JEIF6IFP7OOIB6YKV3IAVLA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:42492->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780234011 (2h41m44.56116739s) [noise=true , address=104.36.82.6:4873 , piece_id=KT3PFAPXSI2R7VKIOWG4BLQRLFYGHOYZWNVKJ2WFLHRUW74UO5ZA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:36250->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780254899 (2h41m44.561188289s) [seg_pos={0 0}, upload=2481]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:33:28.813631340 (2h41m44.594564717s) [upload=2481]: Done uploading segments... Jul 27 00:33:28.813655080 (2h41m44.594588456s) [upload=2481]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.036557331s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..X......X.XX....X...X.................XX....X......XX.X] duration: 30.000746848s FAIL (2025-07-27 00:33:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00077858s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.] duration: 485.356412ms FAIL (2025-07-27 00:33:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 485.38246ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X] duration: 30.037870024s FAIL (2025-07-27 00:33:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.037894755s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 250.181713ms FAIL (2025-07-27 00:33:04 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 250.233376ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.X........X...XX...............................X...........] duration: 40.126898939s FAIL (2025-07-27 00:32:44 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.126919107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 207.555629ms FAIL (2025-07-27 00:32:35 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 207.589308ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031851384s FAIL (2025-07-27 00:32:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== in Jul 27 00:31:32.755531495 (2h39m48.536464871s) [seg_pos={0 0}, upload=2455]: Begin upload segment... Jul 27 00:31:32.755580449 (2h39m48.536513818s) [seg_pos={0 0}, upload=2455]: Done begin upload segment. Jul 27 00:31:32.755604838 (2h39m48.536538217s) [noise=true , address=79.127.205.233:4642 , piece_id=LZ2LUU7D4PDW426J6YIINC6NMHFK5VPVZMFEW45EHKPFCRZKYI4A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.756294030 (2h39m48.537227413s) [noise=true , address=64.49.13.138:4483 , piece_id=ANJOLBU75LIE3AQSATABBYCAPZCKMTVEYOD2BMLBW27CSLJS3PNA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.756439527 (2h39m48.53737291s) [noise=true , address=77.90.31.13:4653 , piece_id=WDAJOMGIR2DNHXLXC72F5W4RLLYQO7QDHYOEWZNJK62AOMBE3SNQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.756688533 (2h39m48.537621918s) [noise=true , address=79.127.205.235:4657 , piece_id=XFEZ2C7PCLMTM2W4VKCS3IR2UKZPQJ7JT3RDRU7XTIH2IREUDX4Q, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.756929663 (2h39m48.537863043s) [noise=true , address=79.127.219.47:4048 , piece_id=NS263QCKRZQJ3EVHZXZJQGT2PTZFMUB7RVSIYF3QQAFLQOLSNEUQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757055495 (2h39m48.537988876s) [noise=true , address=104.36.82.55:4210 , piece_id=NND3WRUQSG7XPAHAWI24NT6CDRYXYBDAH4TC24SIPJ6GER5DCFKA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757311437 (2h39m48.538244817s) [noise=true , address=104.36.82.12:4183 , piece_id=A5MP2V7YTA6WBFW4Q4MIW5A5YW4VRIIHY5N6GRZQLZMEED6C7P3Q, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757350604 (2h39m48.53828398s) [noise=true , address=64.49.13.139:4478 , piece_id=BZC3LTB4TV333O3O3RKWP5A6Y7R7XO7V6YOSQEV72P47BD44ZQVQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757644025 (2h39m48.5385774s) [noise=true , address=79.127.219.38:4009 , piece_id=7KNVGI5BUCDQAX7HN2BCH4JIGJRX2TBYZDCB4TBWKYA6EKGHWX6A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757699116 (2h39m48.538632498s) [noise=true , address=77.90.31.19:4844 , piece_id=VJW6FHTJSS73KP3MYTKS4BMY6R7SVQHJ76XACTHKUGHLRSDETD3A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758010437 (2h39m48.538943815s) [noise=true , address=104.36.82.11:4140 , piece_id=MYE7TBFT5DEIIV5TRLV6QNYDQCPUA7MADXBADHHLCQIC4X74UGVA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758079735 (2h39m48.539013115s) [noise=true , address=79.127.205.234:4620 , piece_id=3RK7LQ46VRABGW4GGH32BEI5NMBECTX7UOHFPRNUFULLJ35CI32A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758377486 (2h39m48.539310866s) [noise=true , address=77.90.31.11:4558 , piece_id=NHI5VU2HKDFVE2C5DPAXNLDPAZAXORXU5BYBFMYSJCPIT3UPWEYQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758495831 (2h39m48.539429208s) [noise=true , address=104.36.82.46:4504 , piece_id=BHBUGIH6KTBP7TJOPYJYQ54HHPJVYZI2GBLJI3UZSRKY3VVQL5YA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758890307 (2h39m48.539823686s) [noise=true , address=77.90.31.15:4727 , piece_id=QEDGCWXWYKXL5HG74VAU7U7XDAICI3H5M7WD2X7BJ5AWZFM6LAMA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.759349209 (2h39m48.540282597s) [noise=true , address=104.36.82.47:4550 , piece_id=D5VN3AY4VJQC7UQ2JX7NUUCMSLQ7FWQQLBPUAATJBTC4CWHVE2TA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.759767619 (2h39m48.540700999s) [noise=true , address=79.127.201.215:4350 , piece_id=FNUF4VR3CMS5YNA57BM3CYRNQFDMNAMCA73AGB7QRKANQGGX3IQQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.760948126 (2h39m48.541881508s) [noise=true , address=156.146.43.225:4448 , piece_id=G4575CWREIA562T2YSTZGJXWMFH3VTXR3CYF4ZCNWGHFC7MYEE7A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.794929087 (2h39m48.575862464s) [noise=true , address=79.127.219.47:4048 , piece_id=NS263QCKRZQJ3EVHZXZJQGT2PTZFMUB7RVSIYF3QQAFLQOLSNEUQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:31:32.832622020 (2h39m48.613555398s) [upload=2455]: Next returned nil segment Jul 27 00:31:32.832646129 (2h39m48.613579517s) [upload=2455]: Waiting for error group managing segments... Jul 27 00:31:32.884669000 (2h39m48.665602494s) [noise=true , address=77.90.31.11:4558 , piece_id=NHI5VU2HKDFVE2C5DPAXNLDPAZAXORXU5BYBFMYSJCPIT3UPWEYQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:32.945001603 (2h39m48.725934984s) [noise=true , address=79.127.219.38:4009 , piece_id=7KNVGI5BUCDQAX7HN2BCH4JIGJRX2TBYZDCB4TBWKYA6EKGHWX6A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:33.350885835 (2h39m49.131819211s) [noise=true , address=79.127.205.233:4642 , piece_id=LZ2LUU7D4PDW426J6YIINC6NMHFK5VPVZMFEW45EHKPFCRZKYI4A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:33.357512910 (2h39m49.138446282s) [noise=true , address=79.127.205.235:4657 , piece_id=XFEZ2C7PCLMTM2W4VKCS3IR2UKZPQJ7JT3RDRU7XTIH2IREUDX4Q, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:33.470874575 (2h39m49.251807954s) [noise=true , address=79.127.205.234:4620 , piece_id=3RK7LQ46VRABGW4GGH32BEI5NMBECTX7UOHFPRNUFULLJ35CI32A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:35.421254041 (2h39m51.202187409s) [noise=true , address=156.146.43.225:4448 , piece_id=G4575CWREIA562T2YSTZGJXWMFH3VTXR3CYF4ZCNWGHFC7MYEE7A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:38.980879644 (2h39m54.761813013s) [noise=true , address=64.49.13.139:4478 , piece_id=BZC3LTB4TV333O3O3RKWP5A6Y7R7XO7V6YOSQEV72P47BD44ZQVQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:39.757612496 (2h39m55.538545887s) [noise=true , address=64.49.13.138:4483 , piece_id=ANJOLBU75LIE3AQSATABBYCAPZCKMTVEYOD2BMLBW27CSLJS3PNA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:46.744699323 (2h40m2.525632704s) [noise=true , address=77.90.31.19:4844 , piece_id=VJW6FHTJSS73KP3MYTKS4BMY6R7SVQHJ76XACTHKUGHLRSDETD3A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:56.624825257 (2h40m12.405758621s) [noise=true , address=79.127.201.215:4350 , piece_id=FNUF4VR3CMS5YNA57BM3CYRNQFDMNAMCA73AGB7QRKANQGGX3IQQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:32:32.680428926 (2h40m48.46136231s) [noise=true , address=77.90.31.13:4653 , piece_id=WDAJOMGIR2DNHXLXC72F5W4RLLYQO7QDHYOEWZNJK62AOMBE3SNQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, address:77.90.31.13:4653): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:32:32.682632774 (2h40m48.463566162s) [noise=true , address=77.90.31.15:4727 , piece_id=QEDGCWXWYKXL5HG74VAU7U7XDAICI3H5M7WD2X7BJ5AWZFM6LAMA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.682738939 (2h40m48.463672303s) [noise=true , address=104.36.82.46:4504 , piece_id=BHBUGIH6KTBP7TJOPYJYQ54HHPJVYZI2GBLJI3UZSRKY3VVQL5YA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: rpc: dial tcp 104.36.82.46:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.682812664 (2h40m48.463746043s) [noise=true , address=104.36.82.12:4183 , piece_id=A5MP2V7YTA6WBFW4Q4MIW5A5YW4VRIIHY5N6GRZQLZMEED6C7P3Q, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: rpc: dial tcp 104.36.82.12:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.682913196 (2h40m48.463846577s) [noise=true , address=104.36.82.11:4140 , piece_id=MYE7TBFT5DEIIV5TRLV6QNYDQCPUA7MADXBADHHLCQIC4X74UGVA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12AYQRkKcRprBgX1BdUwnd4MCQ1swYucsG9aYvLLEaskgrngkCB, address:104.36.82.11:4140): protocol: rpc: dial tcp 104.36.82.11:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.682983903 (2h40m48.463917267s) [noise=true , address=104.36.82.47:4550 , piece_id=D5VN3AY4VJQC7UQ2JX7NUUCMSLQ7FWQQLBPUAATJBTC4CWHVE2TA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.683091255 (2h40m48.464024617s) [noise=true , address=104.36.82.55:4210 , piece_id=NND3WRUQSG7XPAHAWI24NT6CDRYXYBDAH4TC24SIPJ6GER5DCFKA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.683115420 (2h40m48.46404881s) [seg_pos={0 0}, upload=2455]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:32:32.709912829 (2h40m48.490846206s) [upload=2455]: Done uploading segments... Jul 27 00:32:32.709942519 (2h40m48.490875896s) [upload=2455]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.031893925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.] duration: 2.572799671s FAIL (2025-07-27 00:32:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.572853134s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.] duration: 2.511586473s FAIL (2025-07-27 00:32:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 2.129222200 (2h39m57.91015558s) [seg_pos={0 0}, upload=2448]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:31:42.156334822 (2h39m57.937268207s) [upload=2448]: Done uploading segments... Jul 27 00:31:42.156370339 (2h39m57.937303719s) [upload=2448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:31:42.264407036 (2h39m58.045340421s) [upload=2460]: Starting upload Jul 27 00:31:42.264483387 (2h39m58.045416867s) [upload=2460]: Done starting upload Jul 27 00:31:42.264860765 (2h39m58.045794133s) [upload=2460]: Uploading segments... Jul 27 00:31:42.264920808 (2h39m58.045854196s) [upload=2460]: Got next segment. Inline: false Jul 27 00:31:42.264961330 (2h39m58.045894712s) [upload=2460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:42.264966944 (2h39m58.045900329s) [upload=2460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:42.293461093 (2h39m58.074394482s) [seg_pos={0 0}, upload=2460]: Begin upload segment... Jul 27 00:31:42.293548643 (2h39m58.074482013s) [seg_pos={0 0}, upload=2460]: Done begin upload segment. Jul 27 00:31:42.293578727 (2h39m58.074512117s) [noise=true , address=104.36.82.8:4970 , piece_id=FJB7JFLMXUPZPJO3BSTII4TVDSXTDABXK7IHEBEWUPY2FXMECBBQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.293927422 (2h39m58.074860811s) [noise=true , address=45.91.116.4:4939 , piece_id=EGG7N4M3YRMYYLLCSYKD6EIM2VY2FGIQCXEHQ2UNZNMEOQYIZK7A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294124034 (2h39m58.075057425s) [noise=true , address=45.91.116.12:4226 , piece_id=UE4KGLQZDEBRZGTW5WGZQOVOKRI75KYWGEXLVWLSN7Z5ASFDJGBA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294737409 (2h39m58.075670798s) [noise=true , address=45.91.116.7:4009 , piece_id=3HPGGU324DPDQXYG2SDEHWA4Y6DWAE7BX6QJ6D5ZZTINEMFCWH5Q, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294823688 (2h39m58.075757082s) [noise=true , address=77.90.31.18:4807 , piece_id=6UJ4IVWW4KOM2TWHNOO2KJRL4TTNLM2DO6NOCBRCKMRLPQYXJANQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295336032 (2h39m58.07626942s) [noise=true , address=45.91.116.5:4298 , piece_id=6N6Q5ISLB3QUX3DVZ6EW2MWW2XMPAQNV3YB2WVQSFQ2HVWVNE3YA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295509053 (2h39m58.076442443s) [noise=true , address=45.91.116.9:4263 , piece_id=S2SVSBKEGD55RM674IDLJEFUZNEPFK6PGRTQHLHM6XT4WHBE53TA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295753727 (2h39m58.076687124s) [noise=true , address=104.36.82.2:4731 , piece_id=CXTVMN6DVKMP3SBREPOBRYEB6H653XWX4GL2QA7SRXSZKL7EGM5A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296251385 (2h39m58.077184773s) [noise=true , address=45.91.116.10:4083 , piece_id=VGFK6IU3365CATKBV7HNAC7VOH3LEHQKOBKHNMJCQZS44ZRNY45A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296332400 (2h39m58.077265787s) [noise=true , address=104.36.82.13:4186 , piece_id=SJS6IHJDMKBYQ4LPF6EDTLZYUI6C4FZVXCTCUDD2MAUKQ4E6JJ6Q, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296643664 (2h39m58.077577054s) [noise=true , address=77.90.31.15:4699 , piece_id=WHASPWOMGQSJK77UCQKAPVWXIA2TNBH7SIU5RPAOAK4AEWNE6YSQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296847000 (2h39m58.077780386s) [noise=true , address=45.91.116.7:4010 , piece_id=Y2EAWRYA2ENUMYMR2KCZLMYAW47LWDHQSJFOXQPAQRQVMABGLHZQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297023987 (2h39m58.077957376s) [noise=true , address=104.36.82.6:4874 , piece_id=L26EDQYIMM7M25AWGNFSNEINT73ZAEPJPX5VHBBPTDPUUCIW7NGA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297350006 (2h39m58.078283395s) [noise=true , address=45.91.116.7:4011 , piece_id=3TSX42H2AR4Y6GKFQ3GO6MYL5RV3L4Q6M4IWYVB6EJ3ZPR4FVOSA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297947360 (2h39m58.078880751s) [noise=true , address=104.36.82.8:4972 , piece_id=G3454LTJPR5LFBE6ZRYWLNIVIHWKVCAGLONNOEATLQX4GNHDV3RQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.298453984 (2h39m58.079387378s) [noise=true , address=104.36.82.10:4079 , piece_id=ZT3WYT65FBQF2OLOARYQTUCUNMT2M2KPHTAUAREMRPTZ3J5Q5CWA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.408836511 (2h39m58.189769874s) [upload=2460]: Next returned nil segment Jul 27 00:31:42.408888035 (2h39m58.189821415s) [upload=2460]: Waiting for error group managing segments... Jul 27 00:31:42.417174109 (2h39m58.198107473s) [noise=true , address=45.91.116.7:4010 , piece_id=Y2EAWRYA2ENUMYMR2KCZLMYAW47LWDHQSJFOXQPAQRQVMABGLHZQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.417620922 (2h39m58.198554287s) [noise=true , address=45.91.116.9:4263 , piece_id=S2SVSBKEGD55RM674IDLJEFUZNEPFK6PGRTQHLHM6XT4WHBE53TA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.418285088 (2h39m58.199218454s) [noise=true , address=45.91.116.5:4298 , piece_id=6N6Q5ISLB3QUX3DVZ6EW2MWW2XMPAQNV3YB2WVQSFQ2HVWVNE3YA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.418846469 (2h39m58.199779834s) [noise=true , address=45.91.116.4:4939 , piece_id=EGG7N4M3YRMYYLLCSYKD6EIM2VY2FGIQCXEHQ2UNZNMEOQYIZK7A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.419322117 (2h39m58.20025548s) [noise=true , address=45.91.116.12:4226 , piece_id=UE4KGLQZDEBRZGTW5WGZQOVOKRI75KYWGEXLVWLSN7Z5ASFDJGBA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.419928318 (2h39m58.200861684s) [noise=true , address=45.91.116.7:4011 , piece_id=3TSX42H2AR4Y6GKFQ3GO6MYL5RV3L4Q6M4IWYVB6EJ3ZPR4FVOSA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.420340054 (2h39m58.201273418s) [noise=true , address=45.91.116.7:4009 , piece_id=3HPGGU324DPDQXYG2SDEHWA4Y6DWAE7BX6QJ6D5ZZTINEMFCWH5Q, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.420923309 (2h39m58.201856673s) [noise=true , address=45.91.116.10:4083 , piece_id=VGFK6IU3365CATKBV7HNAC7VOH3LEHQKOBKHNMJCQZS44ZRNY45A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:43.937493714 (2h39m59.718427082s) [noise=true , address=77.90.31.18:4807 , piece_id=6UJ4IVWW4KOM2TWHNOO2KJRL4TTNLM2DO6NOCBRCKMRLPQYXJANQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:32:12.221890828 (2h40m28.002824206s) [noise=true , address=104.36.82.10:4079 , piece_id=ZT3WYT65FBQF2OLOARYQTUCUNMT2M2KPHTAUAREMRPTZ3J5Q5CWA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.221959112 (2h40m28.002892501s) [noise=true , address=104.36.82.6:4874 , piece_id=L26EDQYIMM7M25AWGNFSNEINT73ZAEPJPX5VHBBPTDPUUCIW7NGA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222019962 (2h40m28.002953345s) [noise=true , address=77.90.31.15:4699 , piece_id=WHASPWOMGQSJK77UCQKAPVWXIA2TNBH7SIU5RPAOAK4AEWNE6YSQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 159.223.143.39:42112->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222065994 (2h40m28.002999372s) [noise=true , address=104.36.82.2:4731 , piece_id=CXTVMN6DVKMP3SBREPOBRYEB6H653XWX4GL2QA7SRXSZKL7EGM5A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222138243 (2h40m28.003071624s) [noise=true , address=104.36.82.13:4186 , piece_id=SJS6IHJDMKBYQ4LPF6EDTLZYUI6C4FZVXCTCUDD2MAUKQ4E6JJ6Q, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222168712 (2h40m28.00310208s) [noise=true , address=104.36.82.8:4970 , piece_id=FJB7JFLMXUPZPJO3BSTII4TVDSXTDABXK7IHEBEWUPY2FXMECBBQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:32:12.222204212 (2h40m28.003137582s) [noise=true , address=104.36.82.8:4972 , piece_id=G3454LTJPR5LFBE6ZRYWLNIVIHWKVCAGLONNOEATLQX4GNHDV3RQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:51288->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222228959 (2h40m28.003162347s) [seg_pos={0 0}, upload=2460]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:32:12.247747435 (2h40m28.028680803s) [upload=2460]: Done uploading segments... Jul 27 00:32:12.247771546 (2h40m28.028704911s) [upload=2460]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.511610751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX] duration: 30.027978284s FAIL (2025-07-27 00:32:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 2.129222200 (2h39m57.91015558s) [seg_pos={0 0}, upload=2448]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:31:42.156334822 (2h39m57.937268207s) [upload=2448]: Done uploading segments... Jul 27 00:31:42.156370339 (2h39m57.937303719s) [upload=2448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:31:42.264407036 (2h39m58.045340421s) [upload=2460]: Starting upload Jul 27 00:31:42.264483387 (2h39m58.045416867s) [upload=2460]: Done starting upload Jul 27 00:31:42.264860765 (2h39m58.045794133s) [upload=2460]: Uploading segments... Jul 27 00:31:42.264920808 (2h39m58.045854196s) [upload=2460]: Got next segment. Inline: false Jul 27 00:31:42.264961330 (2h39m58.045894712s) [upload=2460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:42.264966944 (2h39m58.045900329s) [upload=2460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:42.293461093 (2h39m58.074394482s) [seg_pos={0 0}, upload=2460]: Begin upload segment... Jul 27 00:31:42.293548643 (2h39m58.074482013s) [seg_pos={0 0}, upload=2460]: Done begin upload segment. Jul 27 00:31:42.293578727 (2h39m58.074512117s) [noise=true , address=104.36.82.8:4970 , piece_id=FJB7JFLMXUPZPJO3BSTII4TVDSXTDABXK7IHEBEWUPY2FXMECBBQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.293927422 (2h39m58.074860811s) [noise=true , address=45.91.116.4:4939 , piece_id=EGG7N4M3YRMYYLLCSYKD6EIM2VY2FGIQCXEHQ2UNZNMEOQYIZK7A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294124034 (2h39m58.075057425s) [noise=true , address=45.91.116.12:4226 , piece_id=UE4KGLQZDEBRZGTW5WGZQOVOKRI75KYWGEXLVWLSN7Z5ASFDJGBA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294737409 (2h39m58.075670798s) [noise=true , address=45.91.116.7:4009 , piece_id=3HPGGU324DPDQXYG2SDEHWA4Y6DWAE7BX6QJ6D5ZZTINEMFCWH5Q, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294823688 (2h39m58.075757082s) [noise=true , address=77.90.31.18:4807 , piece_id=6UJ4IVWW4KOM2TWHNOO2KJRL4TTNLM2DO6NOCBRCKMRLPQYXJANQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295336032 (2h39m58.07626942s) [noise=true , address=45.91.116.5:4298 , piece_id=6N6Q5ISLB3QUX3DVZ6EW2MWW2XMPAQNV3YB2WVQSFQ2HVWVNE3YA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295509053 (2h39m58.076442443s) [noise=true , address=45.91.116.9:4263 , piece_id=S2SVSBKEGD55RM674IDLJEFUZNEPFK6PGRTQHLHM6XT4WHBE53TA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295753727 (2h39m58.076687124s) [noise=true , address=104.36.82.2:4731 , piece_id=CXTVMN6DVKMP3SBREPOBRYEB6H653XWX4GL2QA7SRXSZKL7EGM5A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296251385 (2h39m58.077184773s) [noise=true , address=45.91.116.10:4083 , piece_id=VGFK6IU3365CATKBV7HNAC7VOH3LEHQKOBKHNMJCQZS44ZRNY45A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296332400 (2h39m58.077265787s) [noise=true , address=104.36.82.13:4186 , piece_id=SJS6IHJDMKBYQ4LPF6EDTLZYUI6C4FZVXCTCUDD2MAUKQ4E6JJ6Q, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296643664 (2h39m58.077577054s) [noise=true , address=77.90.31.15:4699 , piece_id=WHASPWOMGQSJK77UCQKAPVWXIA2TNBH7SIU5RPAOAK4AEWNE6YSQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296847000 (2h39m58.077780386s) [noise=true , address=45.91.116.7:4010 , piece_id=Y2EAWRYA2ENUMYMR2KCZLMYAW47LWDHQSJFOXQPAQRQVMABGLHZQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297023987 (2h39m58.077957376s) [noise=true , address=104.36.82.6:4874 , piece_id=L26EDQYIMM7M25AWGNFSNEINT73ZAEPJPX5VHBBPTDPUUCIW7NGA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297350006 (2h39m58.078283395s) [noise=true , address=45.91.116.7:4011 , piece_id=3TSX42H2AR4Y6GKFQ3GO6MYL5RV3L4Q6M4IWYVB6EJ3ZPR4FVOSA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297947360 (2h39m58.078880751s) [noise=true , address=104.36.82.8:4972 , piece_id=G3454LTJPR5LFBE6ZRYWLNIVIHWKVCAGLONNOEATLQX4GNHDV3RQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.298453984 (2h39m58.079387378s) [noise=true , address=104.36.82.10:4079 , piece_id=ZT3WYT65FBQF2OLOARYQTUCUNMT2M2KPHTAUAREMRPTZ3J5Q5CWA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.408836511 (2h39m58.189769874s) [upload=2460]: Next returned nil segment Jul 27 00:31:42.408888035 (2h39m58.189821415s) [upload=2460]: Waiting for error group managing segments... Jul 27 00:31:42.417174109 (2h39m58.198107473s) [noise=true , address=45.91.116.7:4010 , piece_id=Y2EAWRYA2ENUMYMR2KCZLMYAW47LWDHQSJFOXQPAQRQVMABGLHZQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.417620922 (2h39m58.198554287s) [noise=true , address=45.91.116.9:4263 , piece_id=S2SVSBKEGD55RM674IDLJEFUZNEPFK6PGRTQHLHM6XT4WHBE53TA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.418285088 (2h39m58.199218454s) [noise=true , address=45.91.116.5:4298 , piece_id=6N6Q5ISLB3QUX3DVZ6EW2MWW2XMPAQNV3YB2WVQSFQ2HVWVNE3YA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.418846469 (2h39m58.199779834s) [noise=true , address=45.91.116.4:4939 , piece_id=EGG7N4M3YRMYYLLCSYKD6EIM2VY2FGIQCXEHQ2UNZNMEOQYIZK7A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.419322117 (2h39m58.20025548s) [noise=true , address=45.91.116.12:4226 , piece_id=UE4KGLQZDEBRZGTW5WGZQOVOKRI75KYWGEXLVWLSN7Z5ASFDJGBA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.419928318 (2h39m58.200861684s) [noise=true , address=45.91.116.7:4011 , piece_id=3TSX42H2AR4Y6GKFQ3GO6MYL5RV3L4Q6M4IWYVB6EJ3ZPR4FVOSA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.420340054 (2h39m58.201273418s) [noise=true , address=45.91.116.7:4009 , piece_id=3HPGGU324DPDQXYG2SDEHWA4Y6DWAE7BX6QJ6D5ZZTINEMFCWH5Q, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.420923309 (2h39m58.201856673s) [noise=true , address=45.91.116.10:4083 , piece_id=VGFK6IU3365CATKBV7HNAC7VOH3LEHQKOBKHNMJCQZS44ZRNY45A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:43.937493714 (2h39m59.718427082s) [noise=true , address=77.90.31.18:4807 , piece_id=6UJ4IVWW4KOM2TWHNOO2KJRL4TTNLM2DO6NOCBRCKMRLPQYXJANQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:32:12.221890828 (2h40m28.002824206s) [noise=true , address=104.36.82.10:4079 , piece_id=ZT3WYT65FBQF2OLOARYQTUCUNMT2M2KPHTAUAREMRPTZ3J5Q5CWA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.221959112 (2h40m28.002892501s) [noise=true , address=104.36.82.6:4874 , piece_id=L26EDQYIMM7M25AWGNFSNEINT73ZAEPJPX5VHBBPTDPUUCIW7NGA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222019962 (2h40m28.002953345s) [noise=true , address=77.90.31.15:4699 , piece_id=WHASPWOMGQSJK77UCQKAPVWXIA2TNBH7SIU5RPAOAK4AEWNE6YSQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 159.223.143.39:42112->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222065994 (2h40m28.002999372s) [noise=true , address=104.36.82.2:4731 , piece_id=CXTVMN6DVKMP3SBREPOBRYEB6H653XWX4GL2QA7SRXSZKL7EGM5A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222138243 (2h40m28.003071624s) [noise=true , address=104.36.82.13:4186 , piece_id=SJS6IHJDMKBYQ4LPF6EDTLZYUI6C4FZVXCTCUDD2MAUKQ4E6JJ6Q, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222168712 (2h40m28.00310208s) [noise=true , address=104.36.82.8:4970 , piece_id=FJB7JFLMXUPZPJO3BSTII4TVDSXTDABXK7IHEBEWUPY2FXMECBBQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:32:12.222204212 (2h40m28.003137582s) [noise=true , address=104.36.82.8:4972 , piece_id=G3454LTJPR5LFBE6ZRYWLNIVIHWKVCAGLONNOEATLQX4GNHDV3RQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:51288->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222228959 (2h40m28.003162347s) [seg_pos={0 0}, upload=2460]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:32:12.247747435 (2h40m28.028680803s) [upload=2460]: Done uploading segments... Jul 27 00:32:12.247771546 (2h40m28.028704911s) [upload=2460]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.027999544s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X.X........X...XX...............................X..........] duration: 58.11963676s FAIL (2025-07-27 00:32:04 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 58.119663186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X..X......X.XX....X...X.................XX....X......XX.] duration: 2.096086197s FAIL (2025-07-27 00:32:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.096142657s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X..X......X.XX....X...X.................XX....X......XX] duration: 30.001743201s FAIL (2025-07-27 00:31:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001765835s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [X.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX] duration: 30.03478209s FAIL (2025-07-27 00:31:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.034802587s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..X] duration: 30.032623612s FAIL (2025-07-27 00:31:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 (2h38m27.729427932s) [noise=true , address=104.36.82.2:4732 , piece_id=FMVOOI66OYTQ5MMLQJDKV33SH5UBOBGH4XVSOIPXUZPHO5FIG7MQ, seg_pos={0 0}, upload=2433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 159.223.143.39:54996->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:11.948517990 (2h38m27.729451357s) [seg_pos={0 0}, upload=2433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:30:11.948618962 (2h38m27.729552332s) [upload=2433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:30:11.948626030 (2h38m27.729559393s) [upload=2433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:30:12.018855324 (2h38m27.799788716s) [upload=2433]: Done uploading segments... Jul 27 00:30:12.018897650 (2h38m27.799831049s) [upload=2433]: Upload finished. err: <nil> Jul 27 00:31:12.159695723 (2h39m27.940629087s) [upload=2448]: Starting upload Jul 27 00:31:12.159767937 (2h39m27.940701312s) [upload=2448]: Done starting upload Jul 27 00:31:12.160068267 (2h39m27.94100163s) [upload=2448]: Uploading segments... Jul 27 00:31:12.160432583 (2h39m27.941365963s) [upload=2448]: Got next segment. Inline: false Jul 27 00:31:12.160520325 (2h39m27.941453712s) [upload=2448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:12.160546955 (2h39m27.941480339s) [upload=2448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:12.197153293 (2h39m27.978086678s) [seg_pos={0 0}, upload=2448]: Begin upload segment... Jul 27 00:31:12.197241485 (2h39m27.978174874s) [noise=true , address=45.91.116.10:4082 , piece_id=2IN4MTZPW6BQ6ZMBJYM45SEBNFMHKDAXKIAZQLJP6PSBQUJ24U2Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.202968821 (2h39m27.98390221s) [seg_pos={0 0}, upload=2448]: Done begin upload segment. Jul 27 00:31:12.203093298 (2h39m27.984026687s) [noise=true , address=77.90.31.14:4662 , piece_id=OU2W5SJZ7NU5ZTJQQHOIPN7LCEAQT5OGBW2CKJ4USKGWKDAYMFKA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.203579409 (2h39m27.984512799s) [noise=true , address=77.90.31.14:4663 , piece_id=FRZPEUEMNTJDX27WADLUBC22CXE74JPSFOUDNNL7NJ4TMCNZMF5A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.204015604 (2h39m27.984948989s) [noise=true , address=45.91.116.12:4225 , piece_id=LFZYMP5GSIOZDCYFHPMY64PLVZC5CBCNK7DO2M77GCL4JKA46TNA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.204401073 (2h39m27.985334462s) [noise=true , address=77.90.31.13:4627 , piece_id=AZSM4QUF77TECDU77PFUKT6YALD6Y4PUJBTJSAHY7UYFX46WLQMA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205013220 (2h39m27.985946607s) [noise=true , address=45.91.116.13:4155 , piece_id=3MJY3OVR34AU53LQFYLYCJ7IQVOGM3SKN6VRSCIZF6XHVDKJQI3Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205444844 (2h39m27.986378234s) [noise=true , address=104.36.82.5:4839 , piece_id=FABXNVRH7NJS5V6OJ3EBQVXBIUTVYPBRQC6OXECMZANGYNWMNMZQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205810422 (2h39m27.986743808s) [noise=true , address=45.91.116.5:4298 , piece_id=M6EQY4PDGWSZOC7SXMEJXT3FGKOH5KLJ4U4L5DECVHXX6HSKNAWA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.206184793 (2h39m27.987118178s) [noise=true , address=77.90.31.15:4698 , piece_id=E24T5XO47376KU65ACGW2VJIOQ57YXKZUN6YVQDPVCQNAG5RZH2A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.206619846 (2h39m27.987553232s) [noise=true , address=45.91.116.5:4299 , piece_id=ODBKEKETYG7L3UKMXYTFJAINZIOHG4YGDUSXM6JKMBDYKI4VFEXA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.207165081 (2h39m27.988098471s) [noise=true , address=45.91.116.14:4191 , piece_id=PO2GPLLZU7CQ3AH7KK57TRRZPAPX26JB7B3O4XA3ZO3MNOMK4RKQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.207850036 (2h39m27.988783423s) [noise=true , address=104.36.82.13:4185 , piece_id=WRNHR7X33VHZMMDCLHBIJFXU3HYNJT5FLATA34JMOZFNIC5ATDLQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.209818931 (2h39m27.990752331s) [noise=true , address=45.91.116.6:4973 , piece_id=PJ6SFQ37O525LBTWMFVXPUTZODL43XO5O5X7PCCZ4ONKBZNTQL7Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.210267025 (2h39m27.991200412s) [noise=true , address=45.91.116.8:4045 , piece_id=4GLSLSJ4SEYP6Z4LUZKC3UUT44KEZXUJRTMGVR23DKLV5M3PIK5A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.210661756 (2h39m27.991595142s) [noise=true , address=104.36.82.8:4969 , piece_id=CA667UCM3VDDIPWLRZ6URFSSJOJXMUAQX4KJWJSYLL3DJBXWK7CA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.214500450 (2h39m27.995433836s) [noise=true , address=77.90.31.11:4553 , piece_id=NFR6VI445RLIQ7AO6GCEMASE5MKGQYHMMRWJCCLNORO3C6YIXRXA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.322882780 (2h39m28.103816164s) [upload=2448]: Next returned nil segment Jul 27 00:31:12.322902688 (2h39m28.103836074s) [upload=2448]: Waiting for error group managing segments... Jul 27 00:31:12.327209057 (2h39m28.108142436s) [noise=true , address=45.91.116.5:4299 , piece_id=ODBKEKETYG7L3UKMXYTFJAINZIOHG4YGDUSXM6JKMBDYKI4VFEXA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.328683217 (2h39m28.109616595s) [noise=true , address=45.91.116.6:4973 , piece_id=PJ6SFQ37O525LBTWMFVXPUTZODL43XO5O5X7PCCZ4ONKBZNTQL7Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.329209607 (2h39m28.110142971s) [noise=true , address=45.91.116.12:4225 , piece_id=LFZYMP5GSIOZDCYFHPMY64PLVZC5CBCNK7DO2M77GCL4JKA46TNA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.330333791 (2h39m28.111267177s) [noise=true , address=77.90.31.14:4662 , piece_id=OU2W5SJZ7NU5ZTJQQHOIPN7LCEAQT5OGBW2CKJ4USKGWKDAYMFKA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.331153586 (2h39m28.112086957s) [noise=true , address=45.91.116.10:4082 , piece_id=2IN4MTZPW6BQ6ZMBJYM45SEBNFMHKDAXKIAZQLJP6PSBQUJ24U2Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.332133429 (2h39m28.113066797s) [noise=true , address=77.90.31.14:4663 , piece_id=FRZPEUEMNTJDX27WADLUBC22CXE74JPSFOUDNNL7NJ4TMCNZMF5A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.332833107 (2h39m28.113766485s) [noise=true , address=45.91.116.8:4045 , piece_id=4GLSLSJ4SEYP6Z4LUZKC3UUT44KEZXUJRTMGVR23DKLV5M3PIK5A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.333485917 (2h39m28.11441928s) [noise=true , address=45.91.116.5:4298 , piece_id=M6EQY4PDGWSZOC7SXMEJXT3FGKOH5KLJ4U4L5DECVHXX6HSKNAWA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.334923028 (2h39m28.115856408s) [noise=true , address=45.91.116.13:4155 , piece_id=3MJY3OVR34AU53LQFYLYCJ7IQVOGM3SKN6VRSCIZF6XHVDKJQI3Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.377488636 (2h39m28.158422001s) [noise=true , address=45.91.116.14:4191 , piece_id=PO2GPLLZU7CQ3AH7KK57TRRZPAPX26JB7B3O4XA3ZO3MNOMK4RKQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.442882082 (2h39m28.22381546s) [noise=true , address=77.90.31.11:4553 , piece_id=NFR6VI445RLIQ7AO6GCEMASE5MKGQYHMMRWJCCLNORO3C6YIXRXA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:13.842044043 (2h39m29.62297741s) [noise=true , address=77.90.31.13:4627 , piece_id=AZSM4QUF77TECDU77PFUKT6YALD6Y4PUJBTJSAHY7UYFX46WLQMA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:42.128926436 (2h39m57.909859814s) [noise=true , address=104.36.82.5:4839 , piece_id=FABXNVRH7NJS5V6OJ3EBQVXBIUTVYPBRQC6OXECMZANGYNWMNMZQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129053492 (2h39m57.909986871s) [noise=true , address=104.36.82.13:4185 , piece_id=WRNHR7X33VHZMMDCLHBIJFXU3HYNJT5FLATA34JMOZFNIC5ATDLQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129130004 (2h39m57.910063384s) [noise=true , address=104.36.82.8:4969 , piece_id=CA667UCM3VDDIPWLRZ6URFSSJOJXMUAQX4KJWJSYLL3DJBXWK7CA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129201387 (2h39m57.910134767s) [noise=true , address=77.90.31.15:4698 , piece_id=E24T5XO47376KU65ACGW2VJIOQ57YXKZUN6YVQDPVCQNAG5RZH2A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 159.223.143.39:40516->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129222200 (2h39m57.91015558s) [seg_pos={0 0}, upload=2448]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:31:42.156334822 (2h39m57.937268207s) [upload=2448]: Done uploading segments... Jul 27 00:31:42.156370339 (2h39m57.937303719s) [upload=2448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.032649518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 223.567255ms FAIL (2025-07-27 00:31:36 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 223.60094ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029963095s FAIL (2025-07-27 00:31:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Inline: false Jul 27 00:30:26.724536733 (2h38m42.505470119s) [upload=2437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:30:26.724543456 (2h38m42.505476837s) [upload=2437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:30:26.759088225 (2h38m42.540021593s) [seg_pos={0 0}, upload=2437]: Begin upload segment... Jul 27 00:30:26.759219093 (2h38m42.540152474s) [seg_pos={0 0}, upload=2437]: Done begin upload segment. Jul 27 00:30:26.759245235 (2h38m42.540178624s) [noise=true , address=77.90.31.21:4888 , piece_id=3EZ5Y5VMPSQJJDHKSBFYJPKPMTYNL2TIJMZLXE3KQNMN4IITFKIQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.759261652 (2h38m42.540195038s) [noise=true , address=77.90.31.12:4617 , piece_id=G5M3FQUB6XYM5JST2XDUWJLUT7AEQ445AGIS3C7UIJVUNF6SW2UQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.759324596 (2h38m42.540257959s) [noise=true , address=79.127.219.38:4003 , piece_id=HQMHFFMF5RTMY6745SMZMWONAUZHOC6BN22CFZ2TFV2PDQYPMULQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760005812 (2h38m42.5409392s) [noise=true , address=79.127.220.97:4361 , piece_id=CFYELNCF25IV7UH7DZQ44FORQNPN3AUDYUWZTMI4Y2MKNQYZ3BJQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760022149 (2h38m42.540955534s) [noise=true , address=156.146.43.225:4434 , piece_id=NR64SDAEX7AV2JSAPI53NHWJTCDS2Z4JUGBBNR7ZRLA4O56LMEGQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760072046 (2h38m42.541005414s) [noise=true , address=104.36.82.46:4482 , piece_id=2ZDWBSBPM5ZYALK2LLM7ZS52CAF6SELSMGRFFQRXVGEWKDPCRJPA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760523955 (2h38m42.541457343s) [noise=true , address=77.90.31.16:4759 , piece_id=YBKDJVPNOEJS7OES46CDLEGMEITIZXFEI4M72ZYQX27J4Q3MTOSQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760583534 (2h38m42.541516922s) [noise=true , address=79.127.201.215:4349 , piece_id=MW2CVLRLZT3O3XETRD5XJXJX3QGURHTXRMSBST4MZ3KVVV34PCRQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761023884 (2h38m42.541957271s) [noise=true , address=104.36.82.47:4542 , piece_id=YLHHFQFRQ777FOACE2ZWJ4W555GKMJ75QO4IITBVLCHM77235KBA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761074731 (2h38m42.542008122s) [noise=true , address=77.90.31.13:4636 , piece_id=Q445H2SJ6ZEOI5GLWLPAWPYEYS6K4X5BP5HJBVSCDKK3ICEQF4BA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761431108 (2h38m42.54236449s) [noise=true , address=77.90.31.19:4871 , piece_id=HGGMRVOTBRBKCKUW2KBIHVXJEQBVNKGT5XD7AMY4KQRDSOU2L4XA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761563385 (2h38m42.542496775s) [noise=true , address=77.90.31.18:4822 , piece_id=YSR3AOJMALHQXCRAAS7HCGMYRBV2RENDMU3S4QN4NBWIIYBZ3XTQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761921631 (2h38m42.542855017s) [noise=true , address=64.49.13.138:4488 , piece_id=WCVTP7RPSDFOAY7BTQCQRUNFUUHENQTHAXSAQW6KS5JBE73RIORA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.762057081 (2h38m42.542990472s) [noise=true , address=45.91.116.10:4093 , piece_id=YJWWVJFESPJ3D4EJ5WEREFCUGJPOIFTIFDDBB5UZELEEBMTA4GYA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.762305013 (2h38m42.543238402s) [noise=true , address=104.36.82.53:4128 , piece_id=SHXGTCA35HBFRIIRA26SNBUNG75PQ5AW4QP23V7OKXAFYA6BS6OQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.762653757 (2h38m42.543587145s) [noise=true , address=104.36.82.56:4229 , piece_id=KWMKQB6ZKAZX737Y5DHDUWHIQD3AJSHSP4MZKDX3T2WN6PK5ZQJA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.763559754 (2h38m42.54449312s) [noise=true , address=104.36.82.54:4159 , piece_id=QCG75UKHQ345V3OTOBAUVVE6ORLCLND5AS5L77TU7KVSL6TZZ2TA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.766260892 (2h38m42.54719428s) [noise=true , address=104.36.82.19:4432 , piece_id=NZVRAYJXBDVW47VUGNPYP4B2XQ4NUB4I42555ZB35HAR6NT4FJFQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.857899016 (2h38m42.638832381s) [upload=2437]: Next returned nil segment Jul 27 00:30:26.857916404 (2h38m42.638849772s) [upload=2437]: Waiting for error group managing segments... Jul 27 00:30:26.895736919 (2h38m42.676670282s) [noise=true , address=79.127.219.38:4003 , piece_id=HQMHFFMF5RTMY6745SMZMWONAUZHOC6BN22CFZ2TFV2PDQYPMULQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:26.901797700 (2h38m42.682731078s) [noise=true , address=77.90.31.16:4759 , piece_id=YBKDJVPNOEJS7OES46CDLEGMEITIZXFEI4M72ZYQX27J4Q3MTOSQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:26.941352483 (2h38m42.722285845s) [noise=true , address=77.90.31.21:4888 , piece_id=3EZ5Y5VMPSQJJDHKSBFYJPKPMTYNL2TIJMZLXE3KQNMN4IITFKIQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:26.970728884 (2h38m42.751662254s) [noise=true , address=77.90.31.18:4822 , piece_id=YSR3AOJMALHQXCRAAS7HCGMYRBV2RENDMU3S4QN4NBWIIYBZ3XTQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:27.193091405 (2h38m42.974024785s) [noise=true , address=45.91.116.10:4093 , piece_id=YJWWVJFESPJ3D4EJ5WEREFCUGJPOIFTIFDDBB5UZELEEBMTA4GYA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:27.704633744 (2h38m43.485567115s) [noise=true , address=79.127.220.97:4361 , piece_id=CFYELNCF25IV7UH7DZQ44FORQNPN3AUDYUWZTMI4Y2MKNQYZ3BJQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:27.992617758 (2h38m43.773551122s) [noise=true , address=64.49.13.138:4488 , piece_id=WCVTP7RPSDFOAY7BTQCQRUNFUUHENQTHAXSAQW6KS5JBE73RIORA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:28.216496748 (2h38m43.99743013s) [noise=true , address=156.146.43.225:4434 , piece_id=NR64SDAEX7AV2JSAPI53NHWJTCDS2Z4JUGBBNR7ZRLA4O56LMEGQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:29.463318792 (2h38m45.244252186s) [noise=true , address=77.90.31.13:4636 , piece_id=Q445H2SJ6ZEOI5GLWLPAWPYEYS6K4X5BP5HJBVSCDKK3ICEQF4BA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:30.282690166 (2h38m46.06362353s) [noise=true , address=77.90.31.19:4871 , piece_id=HGGMRVOTBRBKCKUW2KBIHVXJEQBVNKGT5XD7AMY4KQRDSOU2L4XA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:35.765953281 (2h38m51.546886648s) [noise=true , address=79.127.201.215:4349 , piece_id=MW2CVLRLZT3O3XETRD5XJXJX3QGURHTXRMSBST4MZ3KVVV34PCRQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:52.798238786 (2h39m8.579172172s) [noise=true , address=77.90.31.12:4617 , piece_id=G5M3FQUB6XYM5JST2XDUWJLUT7AEQ445AGIS3C7UIJVUNF6SW2UQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:31:26.681213804 (2h39m42.462147182s) [noise=true , address=104.36.82.53:4128 , piece_id=SHXGTCA35HBFRIIRA26SNBUNG75PQ5AW4QP23V7OKXAFYA6BS6OQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: rpc: dial tcp 104.36.82.53:4128: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681303735 (2h39m42.4622371s) [noise=true , address=104.36.82.46:4482 , piece_id=2ZDWBSBPM5ZYALK2LLM7ZS52CAF6SELSMGRFFQRXVGEWKDPCRJPA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681371926 (2h39m42.462305292s) [noise=true , address=104.36.82.47:4542 , piece_id=YLHHFQFRQ777FOACE2ZWJ4W555GKMJ75QO4IITBVLCHM77235KBA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: write tcp 159.223.143.39:60858->104.36.82.47:4542: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681449275 (2h39m42.462382636s) [noise=true , address=104.36.82.19:4432 , piece_id=NZVRAYJXBDVW47VUGNPYP4B2XQ4NUB4I42555ZB35HAR6NT4FJFQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:12oh9XZaiLhaZdMyq1cx88yrrG9C9sHtH3RTA6W2Ly39t23ArSQ, address:104.36.82.19:4432): protocol: rpc: dial tcp 104.36.82.19:4432: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681497320 (2h39m42.462430682s) [noise=true , address=104.36.82.56:4229 , piece_id=KWMKQB6ZKAZX737Y5DHDUWHIQD3AJSHSP4MZKDX3T2WN6PK5ZQJA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: rpc: dial tcp 104.36.82.56:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681691579 (2h39m42.46262496s) [noise=true , address=104.36.82.54:4159 , piece_id=QCG75UKHQ345V3OTOBAUVVE6ORLCLND5AS5L77TU7KVSL6TZZ2TA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:104.36.82.54:4159): protocol: write tcp 159.223.143.39:58284->104.36.82.54:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:31:26.681761093 (2h39m42.462694484s) [seg_pos={0 0}, upload=2437]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:31:26.708293966 (2h39m42.489227345s) [upload=2437]: Done uploading segments... Jul 27 00:31:26.708326568 (2h39m42.489259953s) [upload=2437]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.030003126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.X] duration: 30.033702022s FAIL (2025-07-27 00:31:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.033801056s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X..X......X.XX....X...X.................XX....X......X] duration: 30.001978905s FAIL (2025-07-27 00:31:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002011123s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X.X........X...XX...............................X.........] duration: 44.227402898s FAIL (2025-07-27 00:30:57 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.227427659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 251.711027ms FAIL (2025-07-27 00:30:41 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 251.747449ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029713198s FAIL (2025-07-27 00:30:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== :4539 , piece_id=P62QYOR5HDVOXK66SON6ZAPNQ5UC6UQAFYK2OJ3Q4JWIDYUE746A, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.661934430 (2h37m42.442867818s) [noise=true , address=77.90.31.18:4822 , piece_id=4HZEIBK6BQT7U7I2W72FM5O35W7OXBQLL46G6BD3U7KOFD4KBX5Q, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.662314091 (2h37m42.443247474s) [noise=true , address=104.36.82.55:4225 , piece_id=43RTKCHDLS75BHHYKTNKIYCN5YXIORC2SLHSJFUOHVWR5FOJUIWA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.663392193 (2h37m42.444325584s) [noise=true , address=104.36.82.46:4500 , piece_id=QTT4DYSRCSLD2X432VTCGZVFNXXPF4SVCVLZV7UNMFUL6Q3TYUAQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.664032763 (2h37m42.444966149s) [noise=true , address=79.127.205.245:4228 , piece_id=NDBH5VMWRL7GYFNV2SYMS3Q3OT3TDDJNI4QPHVWUF53Y5JT5NQBQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.664450703 (2h37m42.445384143s) [noise=true , address=77.90.31.11:4559 , piece_id=O3MDXB3TP2E4RWDX74VNILW3PAKJBVV5OAE66V45C3KIJYYOTZQA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.664845441 (2h37m42.445778827s) [noise=true , address=79.127.205.234:4603 , piece_id=VNHLUYB532AQPXGY4C3DQ7JK4DRC7YW6RJZND7SDHI2SIPNHWIAA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.665253619 (2h37m42.446187005s) [noise=true , address=77.90.31.17:4780 , piece_id=MI6KK2MS7TT372RYYS2KJB2V4VVWAR6LEXBZVZ7FL5BNKSMMQESA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.665617326 (2h37m42.446550711s) [noise=true , address=79.127.219.38:4010 , piece_id=GVUZO4HFVRUCFZYHEB5JQJ4PG57KOJBW6LPWCYATBIYGDTMIJ5IA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.671138919 (2h37m42.452072308s) [noise=true , address=45.91.116.10:4093 , piece_id=2XHOKOU4JMUKRZV5QOFVZUSGKCMXHQ6DEGQBLKNK66J3EQRRTSVQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.671350640 (2h37m42.452284024s) [noise=true , address=104.36.82.53:4138 , piece_id=AJWKJRMD4ST4TD6B6GL4ZFAM3F6JR32LHHZOPQN53Y437P2PZG4Q, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.671782480 (2h37m42.45271586s) [noise=true , address=45.91.116.4:4964 , piece_id=N6T25Q3JMP4CXHCDEQZMDT5UBOJQYQBXTBEXRIOZ64SD33ZBBULA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.672989070 (2h37m42.453922454s) [noise=true , address=104.36.82.54:4167 , piece_id=P5YVEAEU527VJLXTHS2SYAEEFLZRD5AAAQSR3DRMSMSGEU4GTREA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.673375781 (2h37m42.454309166s) [noise=true , address=45.91.116.14:4206 , piece_id=R4SXHGOIPGCHEQL4ZPWJIPKNCQMIZCU4YCZZBVFGBEJALXCQFMEQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.674160299 (2h37m42.455093712s) [noise=true , address=79.127.201.215:4358 , piece_id=YA3RKYWGE46EWP3WGVBYOEGLHABX3NTWXOAZRIEARO3MZRV27L6Q, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.674679999 (2h37m42.455613387s) [noise=true , address=77.90.31.12:4600 , piece_id=GXAZ6PGFHS2VOMDPCNEHSA5AUJBZK6CVT5C2YCXPKXXAWJUJJD4Q, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.675145437 (2h37m42.456078823s) [noise=true , address=64.49.13.138:4483 , piece_id=OZXYLSNSMIPFEFEPJOVCZCLFZFPC5GAPZ4OMQ6GWBMJGNBH5XEUA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.676664733 (2h37m42.457598121s) [noise=true , address=79.127.219.47:4048 , piece_id=GQ4IITCJE4IWMUMIMTVGTDNMDVQHIENU74YHDJNJJPVGYV4Q4JBQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.782893377 (2h37m42.563826744s) [upload=2421]: Next returned nil segment Jul 27 00:29:26.782915026 (2h37m42.563848414s) [upload=2421]: Waiting for error group managing segments... Jul 27 00:29:26.816774288 (2h37m42.597707669s) [noise=true , address=79.127.219.38:4010 , piece_id=GVUZO4HFVRUCFZYHEB5JQJ4PG57KOJBW6LPWCYATBIYGDTMIJ5IA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:27.097229720 (2h37m42.878163102s) [noise=true , address=79.127.219.47:4048 , piece_id=GQ4IITCJE4IWMUMIMTVGTDNMDVQHIENU74YHDJNJJPVGYV4Q4JBQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:29:27.138066738 (2h37m42.91900011s) [noise=true , address=77.90.31.18:4822 , piece_id=4HZEIBK6BQT7U7I2W72FM5O35W7OXBQLL46G6BD3U7KOFD4KBX5Q, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:27.334775925 (2h37m43.115709293s) [noise=true , address=79.127.205.245:4228 , piece_id=NDBH5VMWRL7GYFNV2SYMS3Q3OT3TDDJNI4QPHVWUF53Y5JT5NQBQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:27.380866201 (2h37m43.161799569s) [noise=true , address=64.49.13.138:4483 , piece_id=OZXYLSNSMIPFEFEPJOVCZCLFZFPC5GAPZ4OMQ6GWBMJGNBH5XEUA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:27.382245435 (2h37m43.163178812s) [noise=true , address=79.127.205.234:4603 , piece_id=VNHLUYB532AQPXGY4C3DQ7JK4DRC7YW6RJZND7SDHI2SIPNHWIAA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:29.241791083 (2h37m45.022724466s) [noise=true , address=77.90.31.12:4600 , piece_id=GXAZ6PGFHS2VOMDPCNEHSA5AUJBZK6CVT5C2YCXPKXXAWJUJJD4Q, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:30.633493226 (2h37m46.41442664s) [noise=true , address=45.91.116.10:4093 , piece_id=2XHOKOU4JMUKRZV5QOFVZUSGKCMXHQ6DEGQBLKNK66J3EQRRTSVQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:32.319644319 (2h37m48.100577696s) [noise=true , address=45.91.116.4:4964 , piece_id=N6T25Q3JMP4CXHCDEQZMDT5UBOJQYQBXTBEXRIOZ64SD33ZBBULA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:44.407666309 (2h38m0.188599691s) [noise=true , address=45.91.116.14:4206 , piece_id=R4SXHGOIPGCHEQL4ZPWJIPKNCQMIZCU4YCZZBVFGBEJALXCQFMEQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:30:01.041240194 (2h38m16.82217357s) [noise=true , address=79.127.201.215:4358 , piece_id=YA3RKYWGE46EWP3WGVBYOEGLHABX3NTWXOAZRIEARO3MZRV27L6Q, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:30:26.583344427 (2h38m42.36427779s) [noise=true , address=104.36.82.54:4167 , piece_id=P5YVEAEU527VJLXTHS2SYAEEFLZRD5AAAQSR3DRMSMSGEU4GTREA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: write tcp 159.223.143.39:43782->104.36.82.54:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.583504686 (2h38m42.364438069s) [noise=true , address=104.36.82.55:4225 , piece_id=43RTKCHDLS75BHHYKTNKIYCN5YXIORC2SLHSJFUOHVWR5FOJUIWA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: write tcp 159.223.143.39:45350->104.36.82.55:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.583663569 (2h38m42.364596957s) [noise=true , address=104.36.82.46:4500 , piece_id=QTT4DYSRCSLD2X432VTCGZVFNXXPF4SVCVLZV7UNMFUL6Q3TYUAQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 159.223.143.39:50636->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.583768554 (2h38m42.364701941s) [noise=true , address=77.90.31.17:4780 , piece_id=MI6KK2MS7TT372RYYS2KJB2V4VVWAR6LEXBZVZ7FL5BNKSMMQESA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: write tcp 159.223.143.39:54900->77.90.31.17:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.583975215 (2h38m42.364908592s) [noise=true , address=104.36.82.47:4539 , piece_id=P62QYOR5HDVOXK66SON6ZAPNQ5UC6UQAFYK2OJ3Q4JWIDYUE746A, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: rpc: dial tcp 104.36.82.47:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.584453239 (2h38m42.365386618s) [noise=true , address=77.90.31.11:4559 , piece_id=O3MDXB3TP2E4RWDX74VNILW3PAKJBVV5OAE66V45C3KIJYYOTZQA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 159.223.143.39:45700->77.90.31.11:4559: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.584473216 (2h38m42.365406583s) [noise=true , address=104.36.82.53:4138 , piece_id=AJWKJRMD4ST4TD6B6GL4ZFAM3F6JR32LHHZOPQN53Y437P2PZG4Q, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: write tcp 159.223.143.39:34800->104.36.82.53:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.584563532 (2h38m42.36549691s) [seg_pos={0 0}, upload=2421]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:30:26.610545833 (2h38m42.391479205s) [upload=2421]: Done uploading segments... Jul 27 00:30:26.610576253 (2h38m42.391509619s) [upload=2421]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.029748812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..] duration: 1.849956926s FAIL (2025-07-27 00:30:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 159.223.143.39:51274->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511244882 (2h36m52.29217826s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:36.511286441 (2h36m52.292219832s) [seg_pos={0 0}, upload=2401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:28:36.543236445 (2h36m52.324169827s) [upload=2401]: Done uploading segments... Jul 27 00:28:36.543304166 (2h36m52.324237549s) [upload=2401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:28:36.635198096 (2h36m52.416131462s) [upload=2408]: Starting upload Jul 27 00:28:36.635282791 (2h36m52.416216158s) [upload=2408]: Done starting upload Jul 27 00:28:36.635390123 (2h36m52.416323499s) [upload=2408]: Uploading segments... Jul 27 00:28:36.635558688 (2h36m52.416492068s) [upload=2408]: Got next segment. Inline: false Jul 27 00:28:36.635615791 (2h36m52.416549176s) [upload=2408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:36.635624207 (2h36m52.416557589s) [upload=2408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:36.671104709 (2h36m52.452038096s) [seg_pos={0 0}, upload=2408]: Begin upload segment... Jul 27 00:28:36.671182809 (2h36m52.452116177s) [seg_pos={0 0}, upload=2408]: Done begin upload segment. Jul 27 00:28:36.671242667 (2h36m52.45217605s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.671699881 (2h36m52.452633265s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672237090 (2h36m52.453170477s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672600157 (2h36m52.453533546s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672929956 (2h36m52.453863336s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673266991 (2h36m52.45420038s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673621446 (2h36m52.454554833s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673960690 (2h36m52.454894077s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676064765 (2h36m52.456998144s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676576481 (2h36m52.45750986s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676952754 (2h36m52.457886136s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677320377 (2h36m52.458253757s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677690801 (2h36m52.458624182s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678050323 (2h36m52.458983704s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678430398 (2h36m52.45936378s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678798637 (2h36m52.459732018s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.774901091 (2h36m52.555834474s) [upload=2408]: Next returned nil segment Jul 27 00:28:36.774921252 (2h36m52.555854639s) [upload=2408]: Waiting for error group managing segments... Jul 27 00:28:36.778853374 (2h36m52.559786751s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.778997842 (2h36m52.559931219s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779321513 (2h36m52.560254881s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779807528 (2h36m52.560740898s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779925920 (2h36m52.560859296s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780369990 (2h36m52.56130336s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780649742 (2h36m52.561583126s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780760157 (2h36m52.561693527s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781178115 (2h36m52.562111492s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781195263 (2h36m52.562128639s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781643788 (2h36m52.562577166s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:38.197748474 (2h36m53.978681856s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:29:06.600686278 (2h37m22.381619658s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:55150->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600775029 (2h37m22.381708408s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:57018->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600786915 (2h37m22.381720287s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56824->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600852455 (2h37m22.381785822s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:57226->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600896768 (2h37m22.381830156s) [seg_pos={0 0}, upload=2408]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:06.629712908 (2h37m22.410646275s) [upload=2408]: Done uploading segments... Jul 27 00:29:06.629739569 (2h37m22.410672946s) [upload=2408]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.849993826s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X.X........X...XX...............................X........] duration: 51.214764286s FAIL (2025-07-27 00:30:12 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.214961767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX.] duration: 2.013216973s FAIL (2025-07-27 00:29:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 159.223.143.39:51274->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511244882 (2h36m52.29217826s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:36.511286441 (2h36m52.292219832s) [seg_pos={0 0}, upload=2401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:28:36.543236445 (2h36m52.324169827s) [upload=2401]: Done uploading segments... Jul 27 00:28:36.543304166 (2h36m52.324237549s) [upload=2401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:28:36.635198096 (2h36m52.416131462s) [upload=2408]: Starting upload Jul 27 00:28:36.635282791 (2h36m52.416216158s) [upload=2408]: Done starting upload Jul 27 00:28:36.635390123 (2h36m52.416323499s) [upload=2408]: Uploading segments... Jul 27 00:28:36.635558688 (2h36m52.416492068s) [upload=2408]: Got next segment. Inline: false Jul 27 00:28:36.635615791 (2h36m52.416549176s) [upload=2408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:36.635624207 (2h36m52.416557589s) [upload=2408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:36.671104709 (2h36m52.452038096s) [seg_pos={0 0}, upload=2408]: Begin upload segment... Jul 27 00:28:36.671182809 (2h36m52.452116177s) [seg_pos={0 0}, upload=2408]: Done begin upload segment. Jul 27 00:28:36.671242667 (2h36m52.45217605s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.671699881 (2h36m52.452633265s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672237090 (2h36m52.453170477s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672600157 (2h36m52.453533546s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672929956 (2h36m52.453863336s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673266991 (2h36m52.45420038s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673621446 (2h36m52.454554833s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673960690 (2h36m52.454894077s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676064765 (2h36m52.456998144s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676576481 (2h36m52.45750986s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676952754 (2h36m52.457886136s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677320377 (2h36m52.458253757s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677690801 (2h36m52.458624182s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678050323 (2h36m52.458983704s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678430398 (2h36m52.45936378s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678798637 (2h36m52.459732018s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.774901091 (2h36m52.555834474s) [upload=2408]: Next returned nil segment Jul 27 00:28:36.774921252 (2h36m52.555854639s) [upload=2408]: Waiting for error group managing segments... Jul 27 00:28:36.778853374 (2h36m52.559786751s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.778997842 (2h36m52.559931219s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779321513 (2h36m52.560254881s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779807528 (2h36m52.560740898s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779925920 (2h36m52.560859296s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780369990 (2h36m52.56130336s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780649742 (2h36m52.561583126s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780760157 (2h36m52.561693527s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781178115 (2h36m52.562111492s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781195263 (2h36m52.562128639s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781643788 (2h36m52.562577166s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:38.197748474 (2h36m53.978681856s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:29:06.600686278 (2h37m22.381619658s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:55150->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600775029 (2h37m22.381708408s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:57018->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600786915 (2h37m22.381720287s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56824->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600852455 (2h37m22.381785822s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:57226->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600896768 (2h37m22.381830156s) [seg_pos={0 0}, upload=2408]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:06.629712908 (2h37m22.410646275s) [upload=2408]: Done uploading segments... Jul 27 00:29:06.629739569 (2h37m22.410672946s) [upload=2408]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.013243864s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X..X......X.XX....X...X.................XX....X......] duration: 1.819044687s FAIL (2025-07-27 00:29:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.819085583s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.] duration: 1.976758114s FAIL (2025-07-27 00:29:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.97680179s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX] duration: 30.042374892s FAIL (2025-07-27 00:29:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.04241149s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 231.015149ms FAIL (2025-07-27 00:29:32 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 231.076638ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.032699292s FAIL (2025-07-27 00:29:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== _pos={0 0}, upload=2406]: Begin upload segment... Jul 27 00:28:26.566373976 (2h36m42.347307348s) [seg_pos={0 0}, upload=2406]: Done begin upload segment. Jul 27 00:28:26.566413627 (2h36m42.347347013s) [noise=true , address=104.36.82.53:4134 , piece_id=JYTJJMDG5NCTTVC46XAMKLLKMZ5P3NIPSB7UWX4BY5W53BUFU4LA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.566936420 (2h36m42.347869808s) [noise=true , address=79.127.205.245:4225 , piece_id=V4OVOLWR6F4H2ILVCEIZA7SPDXDN6XR7AFD5FM3VCXYOLEOKTNKA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.567288935 (2h36m42.348222311s) [noise=true , address=77.90.31.19:4875 , piece_id=GRNGLQ67YDNHPU6BJCQJSCV2AC4HJOUT3LNG545FPZAMYZWGOW2A, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.567426154 (2h36m42.34835954s) [noise=true , address=104.36.82.52:4095 , piece_id=GMRVDC6YFYLISN34VAU5SZALSOBRUQLHUP32WSFHY732Z6XTAONQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.567973431 (2h36m42.348907014s) [noise=true , address=77.90.31.21:4897 , piece_id=LYFCOCMX7O37WMMFDBMYEP5XNBAD7G74DEBMWTX6ZSCVQS6GFS5Q, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568027009 (2h36m42.348960396s) [noise=true , address=45.91.116.8:4062 , piece_id=6HXNRHI72DI7NEBRNDFSKXV4O7BWSMWMFR5T2CYZALMT63H3G2IA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568126507 (2h36m42.349059899s) [noise=true , address=77.90.31.12:4621 , piece_id=6NGMIDF6AK56CTICEYNWNTEW5X7MY6UBONZBZLPX24WF7K4332YQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568579578 (2h36m42.349512969s) [noise=true , address=104.36.82.56:4261 , piece_id=65XTR42O43FJU57DZH4Z6MTUUA2WLBV7KIYTEZ2PVC6ZM3T34BBQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568615436 (2h36m42.349548824s) [noise=true , address=104.36.82.11:4138 , piece_id=U776CGTFPNHFUW6E7LQUCCOGWWZV3LO37EEMTTSPPKTAHLP4E7GQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568998998 (2h36m42.349932436s) [noise=true , address=104.36.82.55:4220 , piece_id=YV4LB7XLT5OPFA7JUSDW4UPAWEN6P34TSKARHZHLPWFUYRFDLVIA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.569016887 (2h36m42.349950274s) [noise=true , address=64.49.13.140:4514 , piece_id=HRW6BF45DA5CB4WRWWE55G3CMS74CDPCIZD6QWHWMEDZKMT42MBQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.569440076 (2h36m42.350373464s) [noise=true , address=79.127.219.38:4004 , piece_id=VXO5WHWP3S4HRTKPAJFCRCFLWFZESFOKFEYPQPQDGSCMCOQWVAGA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.569911732 (2h36m42.350845118s) [noise=true , address=79.127.201.215:4360 , piece_id=LM5GL7CAU2XCGTVCJYXVTDBUHSXBJUPPIAZBSRTWDSBJG7N7MUEQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.571601249 (2h36m42.352534638s) [noise=true , address=64.49.13.138:4483 , piece_id=DRBL3BTL6WRQHUEDPHSSIUOVWMPXDJV4TZUKBFVZK66H4LAPNQYQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.572216434 (2h36m42.353149826s) [noise=true , address=104.36.82.47:4526 , piece_id=ADAI444AWHNQEHBEBAQTDJ25ZPHOVIJAUF736TLBYE2MKL7RIONA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.572709516 (2h36m42.353642904s) [noise=true , address=104.36.82.46:4484 , piece_id=TIUCHXHM5GFIA6VGZWXF6GVSPENIGNWKJSFGR5DT53URBVL5N6GA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.573075818 (2h36m42.354009206s) [noise=true , address=79.127.201.214:4289 , piece_id=MLMZ5DQ3SUD2Z5VDDIAHT22YF5PKMKUYSB2ISV6BFXCX6WOGVX2Q, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.573597876 (2h36m42.354531256s) [noise=true , address=45.91.116.4:4964 , piece_id=5D76VB45ECVXTSHSZDWFHSRFE5GY4NS7D7ZSMLDSKGTQGQQDVF6Q, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.623013151 (2h36m42.403946519s) [noise=true , address=77.90.31.19:4875 , piece_id=GRNGLQ67YDNHPU6BJCQJSCV2AC4HJOUT3LNG545FPZAMYZWGOW2A, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:26.657378540 (2h36m42.438311908s) [upload=2406]: Next returned nil segment Jul 27 00:28:26.657396149 (2h36m42.438329526s) [upload=2406]: Waiting for error group managing segments... Jul 27 00:28:26.673084853 (2h36m42.454018235s) [noise=true , address=79.127.219.38:4004 , piece_id=VXO5WHWP3S4HRTKPAJFCRCFLWFZESFOKFEYPQPQDGSCMCOQWVAGA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:26.703091616 (2h36m42.484024987s) [noise=true , address=77.90.31.21:4897 , piece_id=LYFCOCMX7O37WMMFDBMYEP5XNBAD7G74DEBMWTX6ZSCVQS6GFS5Q, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:26.739229394 (2h36m42.520162772s) [noise=true , address=45.91.116.8:4062 , piece_id=6HXNRHI72DI7NEBRNDFSKXV4O7BWSMWMFR5T2CYZALMT63H3G2IA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:27.209815386 (2h36m42.990748762s) [noise=true , address=79.127.205.245:4225 , piece_id=V4OVOLWR6F4H2ILVCEIZA7SPDXDN6XR7AFD5FM3VCXYOLEOKTNKA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:27.323401171 (2h36m43.104334541s) [noise=true , address=64.49.13.140:4514 , piece_id=HRW6BF45DA5CB4WRWWE55G3CMS74CDPCIZD6QWHWMEDZKMT42MBQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:30.605373476 (2h36m46.386306854s) [noise=true , address=79.127.201.214:4289 , piece_id=MLMZ5DQ3SUD2Z5VDDIAHT22YF5PKMKUYSB2ISV6BFXCX6WOGVX2Q, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:35.281060700 (2h36m51.061994086s) [noise=true , address=45.91.116.4:4964 , piece_id=5D76VB45ECVXTSHSZDWFHSRFE5GY4NS7D7ZSMLDSKGTQGQQDVF6Q, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:36.214527059 (2h36m51.995460441s) [noise=true , address=64.49.13.138:4483 , piece_id=DRBL3BTL6WRQHUEDPHSSIUOVWMPXDJV4TZUKBFVZK66H4LAPNQYQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:57.034334653 (2h37m12.815268038s) [noise=true , address=77.90.31.12:4621 , piece_id=6NGMIDF6AK56CTICEYNWNTEW5X7MY6UBONZBZLPX24WF7K4332YQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:59.491551681 (2h37m15.272485045s) [noise=true , address=79.127.201.215:4360 , piece_id=LM5GL7CAU2XCGTVCJYXVTDBUHSXBJUPPIAZBSRTWDSBJG7N7MUEQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:29:26.493759888 (2h37m42.274693269s) [noise=true , address=104.36.82.55:4220 , piece_id=YV4LB7XLT5OPFA7JUSDW4UPAWEN6P34TSKARHZHLPWFUYRFDLVIA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: rpc: dial tcp 104.36.82.55:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.493884848 (2h37m42.274818216s) [noise=true , address=104.36.82.11:4138 , piece_id=U776CGTFPNHFUW6E7LQUCCOGWWZV3LO37EEMTTSPPKTAHLP4E7GQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 159.223.143.39:53730->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.493956328 (2h37m42.274889706s) [noise=true , address=104.36.82.46:4484 , piece_id=TIUCHXHM5GFIA6VGZWXF6GVSPENIGNWKJSFGR5DT53URBVL5N6GA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: rpc: dial tcp 104.36.82.46:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.494029799 (2h37m42.274963177s) [noise=true , address=104.36.82.52:4095 , piece_id=GMRVDC6YFYLISN34VAU5SZALSOBRUQLHUP32WSFHY732Z6XTAONQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: rpc: dial tcp 104.36.82.52:4095: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.494153785 (2h37m42.275087167s) [noise=true , address=104.36.82.56:4261 , piece_id=65XTR42O43FJU57DZH4Z6MTUUA2WLBV7KIYTEZ2PVC6ZM3T34BBQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: rpc: dial tcp 104.36.82.56:4261: i/o timeout; context deadline exceeded Jul 27 00:29:26.494318860 (2h37m42.275252239s) [noise=true , address=104.36.82.53:4134 , piece_id=JYTJJMDG5NCTTVC46XAMKLLKMZ5P3NIPSB7UWX4BY5W53BUFU4LA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 159.223.143.39:55482->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.494396076 (2h37m42.275329455s) [noise=true , address=104.36.82.47:4526 , piece_id=ADAI444AWHNQEHBEBAQTDJ25ZPHOVIJAUF736TLBYE2MKL7RIONA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: rpc: dial tcp 104.36.82.47:4526: i/o timeout; conn closed: op Jul 27 00:29:26.494455547 (2h37m42.275388985s) [seg_pos={0 0}, upload=2406]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:29:26.524227752 (2h37m42.305161126s) [upload=2406]: Done uploading segments... Jul 27 00:29:26.524260252 (2h37m42.305193629s) [upload=2406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.032724668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X.X........X...XX...............................X.......] duration: 56.870061112s FAIL (2025-07-27 00:29:21 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.870078517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX] duration: 30.031167187s FAIL (2025-07-27 00:29:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 159.223.143.39:51274->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511244882 (2h36m52.29217826s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:36.511286441 (2h36m52.292219832s) [seg_pos={0 0}, upload=2401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:28:36.543236445 (2h36m52.324169827s) [upload=2401]: Done uploading segments... Jul 27 00:28:36.543304166 (2h36m52.324237549s) [upload=2401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:28:36.635198096 (2h36m52.416131462s) [upload=2408]: Starting upload Jul 27 00:28:36.635282791 (2h36m52.416216158s) [upload=2408]: Done starting upload Jul 27 00:28:36.635390123 (2h36m52.416323499s) [upload=2408]: Uploading segments... Jul 27 00:28:36.635558688 (2h36m52.416492068s) [upload=2408]: Got next segment. Inline: false Jul 27 00:28:36.635615791 (2h36m52.416549176s) [upload=2408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:36.635624207 (2h36m52.416557589s) [upload=2408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:36.671104709 (2h36m52.452038096s) [seg_pos={0 0}, upload=2408]: Begin upload segment... Jul 27 00:28:36.671182809 (2h36m52.452116177s) [seg_pos={0 0}, upload=2408]: Done begin upload segment. Jul 27 00:28:36.671242667 (2h36m52.45217605s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.671699881 (2h36m52.452633265s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672237090 (2h36m52.453170477s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672600157 (2h36m52.453533546s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672929956 (2h36m52.453863336s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673266991 (2h36m52.45420038s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673621446 (2h36m52.454554833s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673960690 (2h36m52.454894077s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676064765 (2h36m52.456998144s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676576481 (2h36m52.45750986s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676952754 (2h36m52.457886136s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677320377 (2h36m52.458253757s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677690801 (2h36m52.458624182s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678050323 (2h36m52.458983704s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678430398 (2h36m52.45936378s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678798637 (2h36m52.459732018s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.774901091 (2h36m52.555834474s) [upload=2408]: Next returned nil segment Jul 27 00:28:36.774921252 (2h36m52.555854639s) [upload=2408]: Waiting for error group managing segments... Jul 27 00:28:36.778853374 (2h36m52.559786751s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.778997842 (2h36m52.559931219s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779321513 (2h36m52.560254881s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779807528 (2h36m52.560740898s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779925920 (2h36m52.560859296s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780369990 (2h36m52.56130336s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780649742 (2h36m52.561583126s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780760157 (2h36m52.561693527s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781178115 (2h36m52.562111492s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781195263 (2h36m52.562128639s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781643788 (2h36m52.562577166s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:38.197748474 (2h36m53.978681856s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:29:06.600686278 (2h37m22.381619658s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:55150->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600775029 (2h37m22.381708408s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:57018->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600786915 (2h37m22.381720287s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56824->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600852455 (2h37m22.381785822s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:57226->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600896768 (2h37m22.381830156s) [seg_pos={0 0}, upload=2408]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:06.629712908 (2h37m22.410646275s) [upload=2408]: Done uploading segments... Jul 27 00:29:06.629739569 (2h37m22.410672946s) [upload=2408]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.03119018s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X..X......X.XX....X...X.................XX....X.....] duration: 2.133487573s FAIL (2025-07-27 00:29:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.133516397s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 210.32042ms FAIL (2025-07-27 00:28:57 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 210.350119ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 214.890672ms FAIL (2025-07-27 00:28:40 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 214.918508ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...X] duration: 30.035028715s FAIL (2025-07-27 00:28:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 159.223.143.39:34290->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:50.445416404 (2h36m6.226349786s) [seg_pos={0 0}, upload=2396]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:27:50.445513078 (2h36m6.226446459s) [upload=2396]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:27:50.445518863 (2h36m6.226452227s) [upload=2396]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:27:50.503448342 (2h36m6.284381712s) [upload=2396]: Done uploading segments... Jul 27 00:27:50.503469984 (2h36m6.284403353s) [upload=2396]: Upload finished. err: <nil> Jul 27 00:28:06.550015467 (2h36m22.330948854s) [upload=2401]: Starting upload Jul 27 00:28:06.550183329 (2h36m22.33111669s) [upload=2401]: Done starting upload Jul 27 00:28:06.550579332 (2h36m22.33151271s) [upload=2401]: Uploading segments... Jul 27 00:28:06.550695736 (2h36m22.331629123s) [upload=2401]: Got next segment. Inline: false Jul 27 00:28:06.550739771 (2h36m22.331673154s) [upload=2401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:06.550746924 (2h36m22.331680304s) [upload=2401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:06.603796545 (2h36m22.384729931s) [seg_pos={0 0}, upload=2401]: Begin upload segment... Jul 27 00:28:06.603982778 (2h36m22.384916165s) [seg_pos={0 0}, upload=2401]: Done begin upload segment. Jul 27 00:28:06.604280686 (2h36m22.38521405s) [noise=true , address=77.90.31.19:4841 , piece_id=CCIVOOJRQLM2JO4GXFZIZGLAYZZZ2UGDXU6U3XNEI2BBIP5X4VAA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.604341430 (2h36m22.385274816s) [noise=true , address=45.91.116.11:4117 , piece_id=XQMVHN7RFHG2O3R5I6CFYLKOMFHORZTOLHSREALUSDKS2FQNZMIQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.605853071 (2h36m22.386786453s) [noise=true , address=77.90.31.17:4770 , piece_id=GHZICB4PG7KZ27HWXUZR24OO6IQ3C3TZPZX4G3YVAVKF4G5KLZAA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.606304326 (2h36m22.38723769s) [noise=true , address=104.36.82.5:4838 , piece_id=KVJM5EJUQSB6HY3DTNUX2CG23PSMSHEG7EABKV7BRUUVGUHJAESQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.607588272 (2h36m22.388521634s) [noise=true , address=77.90.31.12:4589 , piece_id=DXRH3VZGO2JX3S6YT5ICRJMDZLRWFXR2R6BZLN5ZL23FP23W2LIA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.608585246 (2h36m22.389518609s) [noise=true , address=45.91.116.9:4263 , piece_id=4YVCO5KY5AT6UP3Z7BWMKACZQ65VUPYIYADJGB423YAX4LGOVOIQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.609659164 (2h36m22.390592542s) [noise=true , address=45.91.116.4:4937 , piece_id=ALBVLHGTQJDNQW5NX542OLZEMTAVC5JFXIYDFHNHO6VXILA4TSOQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.610711285 (2h36m22.391644656s) [noise=true , address=45.91.116.7:4009 , piece_id=CZFVW4PFVOA7MSMNS6IHLE47V4ZRXOLG3EE55ZGRCOJGUZGYIIMQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.612108187 (2h36m22.393041564s) [noise=true , address=104.36.82.16:4293 , piece_id=RRTUXELA7GTJYMOAPAYGZIB2DQYQ6G7LJSZI5KXCUMTPSC46FYSQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.615584609 (2h36m22.396517995s) [noise=true , address=77.90.31.14:4662 , piece_id=A4VMZD2WQZ6NCOTDOGRKWCHMMPVW3CVTIWM2R37V65XRZGMNRLLQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.618899846 (2h36m22.399833216s) [noise=true , address=104.36.82.14:4223 , piece_id=RB5FQ7CLN4QHNJWJ42Q3LHYGIRI5WIIIVVYTK4Q6TRX3TSIPSWJQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.622662503 (2h36m22.403595876s) [noise=true , address=45.91.116.4:4938 , piece_id=VOMPXWYLN2JO6XRMRJ7HZOLET6TUIUK2LHMIL7P3VBURFMDJ6MTQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.625082686 (2h36m22.406016057s) [noise=true , address=45.91.116.13:4153 , piece_id=6U7HI7CUE4H6JA5M2DNDOWVPM2DZSL36NFR64VN4QZG2JXKZC3RQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.627696691 (2h36m22.408630057s) [noise=true , address=45.91.116.12:4226 , piece_id=VDFJBBFYLQMMEIWNPPXLJBXATKRXW6W47G6DBQJDXMISZJFK33YQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.628254642 (2h36m22.409188019s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.630529342 (2h36m22.41146271s) [noise=true , address=45.91.116.6:4975 , piece_id=O53JBF4B6CS2BXIKETZPN6ELYFOIEJOGXJQSIO3GVWYZJIYNW4JA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.733959938 (2h36m22.514893319s) [upload=2401]: Next returned nil segment Jul 27 00:28:06.734004093 (2h36m22.514937478s) [upload=2401]: Waiting for error group managing segments... Jul 27 00:28:06.737109419 (2h36m22.518042799s) [noise=true , address=45.91.116.9:4263 , piece_id=4YVCO5KY5AT6UP3Z7BWMKACZQ65VUPYIYADJGB423YAX4LGOVOIQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.737154929 (2h36m22.518088299s) [noise=true , address=45.91.116.11:4117 , piece_id=XQMVHN7RFHG2O3R5I6CFYLKOMFHORZTOLHSREALUSDKS2FQNZMIQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.738142863 (2h36m22.519076247s) [noise=true , address=45.91.116.4:4937 , piece_id=ALBVLHGTQJDNQW5NX542OLZEMTAVC5JFXIYDFHNHO6VXILA4TSOQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.739076853 (2h36m22.520010231s) [noise=true , address=77.90.31.14:4662 , piece_id=A4VMZD2WQZ6NCOTDOGRKWCHMMPVW3CVTIWM2R37V65XRZGMNRLLQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.739552455 (2h36m22.520485819s) [noise=true , address=45.91.116.6:4975 , piece_id=O53JBF4B6CS2BXIKETZPN6ELYFOIEJOGXJQSIO3GVWYZJIYNW4JA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.739591458 (2h36m22.520524821s) [noise=true , address=45.91.116.7:4009 , piece_id=CZFVW4PFVOA7MSMNS6IHLE47V4ZRXOLG3EE55ZGRCOJGUZGYIIMQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.739641896 (2h36m22.52057526s) [noise=true , address=45.91.116.13:4153 , piece_id=6U7HI7CUE4H6JA5M2DNDOWVPM2DZSL36NFR64VN4QZG2JXKZC3RQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.740085966 (2h36m22.521019345s) [noise=true , address=45.91.116.4:4938 , piece_id=VOMPXWYLN2JO6XRMRJ7HZOLET6TUIUK2LHMIL7P3VBURFMDJ6MTQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.740848032 (2h36m22.521781409s) [noise=true , address=45.91.116.12:4226 , piece_id=VDFJBBFYLQMMEIWNPPXLJBXATKRXW6W47G6DBQJDXMISZJFK33YQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:08.153142773 (2h36m23.934076142s) [noise=true , address=77.90.31.19:4841 , piece_id=CCIVOOJRQLM2JO4GXFZIZGLAYZZZ2UGDXU6U3XNEI2BBIP5X4VAA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:08.885905413 (2h36m24.666838786s) [noise=true , address=77.90.31.12:4589 , piece_id=DXRH3VZGO2JX3S6YT5ICRJMDZLRWFXR2R6BZLN5ZL23FP23W2LIA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:36.510937108 (2h36m52.291870476s) [noise=true , address=77.90.31.17:4770 , piece_id=GHZICB4PG7KZ27HWXUZR24OO6IQ3C3TZPZX4G3YVAVKF4G5KLZAA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.510982410 (2h36m52.291915777s) [noise=true , address=104.36.82.14:4223 , piece_id=RB5FQ7CLN4QHNJWJ42Q3LHYGIRI5WIIIVVYTK4Q6TRX3TSIPSWJQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511012383 (2h36m52.291945755s) [noise=true , address=104.36.82.16:4293 , piece_id=RRTUXELA7GTJYMOAPAYGZIB2DQYQ6G7LJSZI5KXCUMTPSC46FYSQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511054847 (2h36m52.291988216s) [noise=true , address=104.36.82.5:4838 , piece_id=KVJM5EJUQSB6HY3DTNUX2CG23PSMSHEG7EABKV7BRUUVGUHJAESQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:51274->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511244882 (2h36m52.29217826s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:36.511286441 (2h36m52.292219832s) [seg_pos={0 0}, upload=2401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:28:36.543236445 (2h36m52.324169827s) [upload=2401]: Done uploading segments... Jul 27 00:28:36.543304166 (2h36m52.324237549s) [upload=2401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.035049769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXX] duration: 30.028810758s FAIL (2025-07-27 00:28:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.02883982s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.039830012s FAIL (2025-07-27 00:28:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== d=7BMKAKFYJSGBTDB64PQSNQLQCUDX64OOJJGAF3RBAYAERIZ3VNXA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.471231100 (2h35m42.252164508s) [noise=true , address=77.90.31.17:4792 , piece_id=I2EFY5REBRIIWNQYGBT5PETJ4PVPUQPQFL27TG23PHOTNI7L4T4Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.471356473 (2h35m42.25228986s) [noise=true , address=104.36.82.54:4163 , piece_id=22FA4PAWJB4FQJ43NSJRN3K6UA6PCDK7CKFHPGGH6TDOFAF7YOXQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.471772528 (2h35m42.252705915s) [noise=true , address=64.49.13.139:4478 , piece_id=7OUHO2VKK4ZIDDQSBPDEG7KEBQP7GO6J2HNTF7N5Z3O22TTRUR3Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.471962598 (2h35m42.252895985s) [noise=true , address=104.36.82.52:4097 , piece_id=43MACCM5RLSY3IXHPFFYNAKJIQY45E6Z24N7TISGAEQSKHZPEVJA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.472268991 (2h35m42.253202376s) [noise=true , address=79.127.205.245:4248 , piece_id=S25F3LV5UVV6UQRB27OSB76OAFVJJCQDC264D5SKIVFIH6B57KEQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.472509494 (2h35m42.253442871s) [noise=true , address=79.127.220.101:4041 , piece_id=MTYQKW3CLES2OMI5WQ3EBYUANNU7T73MPHR57FB7JNH6EBRGRODA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473024889 (2h35m42.253958278s) [noise=true , address=79.127.205.246:4278 , piece_id=JVCYSTU6YKUOPIWYBA2NTVNAI4TTT6N6JMLVCVTR6MM22TDZ5NTA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473104285 (2h35m42.254037675s) [noise=true , address=79.127.219.38:4006 , piece_id=FMHSWNMCOV5IM2XHDKQGULEJQXBGO6QXAW2GS6L6Y7JJ2TZ653DQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473542915 (2h35m42.254476305s) [noise=true , address=104.36.82.46:4507 , piece_id=SHIBRAZNU65Q5EXKZNJZZ6YT4OJPE2OZOQJH6MUD3EHBJEX4LACA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473570465 (2h35m42.254503849s) [noise=true , address=156.146.43.225:4461 , piece_id=B5K343ACIE27RPLTJIVTVE3DWVMRWJEOHWFHBNXKN7A4J3RS2L7Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473959622 (2h35m42.25489301s) [noise=true , address=104.36.82.55:4202 , piece_id=NKVHLWTADQ6ALYP2Z5KBTRGY4YSC272YQXLOUTVWDRUFEYUCSKDA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.474026753 (2h35m42.254960137s) [noise=true , address=104.36.82.47:4545 , piece_id=5HQAGGCDEHVM52TSNSX3O35ELQYNUO7DKHSJLWITZK4Q3KT7ZB6Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.474445495 (2h35m42.255378879s) [noise=true , address=104.36.82.56:4244 , piece_id=U2EIQHP5VR262NR53LOGXVQ7FOAJ5LYWDWFMB5YJRVIIXLXIKNMQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.474862688 (2h35m42.25579607s) [noise=true , address=77.90.31.13:4658 , piece_id=SKR62SSGFEGH43T2UECLP2HFAYRBNMNKTFMI53OKXB63JGGHJ74A, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.474873122 (2h35m42.255807289s) [noise=true , address=79.127.205.233:4642 , piece_id=WGKZBMVCFZH5SCCYCVDZYMBGLM2LW33QK45VYDCUXIYWMZUKVVHQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.478975067 (2h35m42.259908436s) [noise=true , address=79.127.220.97:4361 , piece_id=TXQYNILTXIKK4NAJ65EQ7JB45SS7PKYP7M5OZ5U2RYPVAWZ4476Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.480027718 (2h35m42.260961097s) [noise=true , address=104.36.82.11:4143 , piece_id=VWDZRME2V7OYFIK5EM7FPNHE2RQ4ALAZFEWOOWNTUZ5AGNFAPQMQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.520474031 (2h35m42.301407417s) [noise=true , address=77.90.31.13:4658 , piece_id=SKR62SSGFEGH43T2UECLP2HFAYRBNMNKTFMI53OKXB63JGGHJ74A, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: storage node overloaded, request limit: 2000 Jul 27 00:27:26.554999884 (2h35m42.335933262s) [upload=2392]: Next returned nil segment Jul 27 00:27:26.555095212 (2h35m42.336028598s) [upload=2392]: Waiting for error group managing segments... Jul 27 00:27:26.559224449 (2h35m42.340157816s) [noise=true , address=79.127.219.38:4006 , piece_id=FMHSWNMCOV5IM2XHDKQGULEJQXBGO6QXAW2GS6L6Y7JJ2TZ653DQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:26.575349290 (2h35m42.356282673s) [noise=true , address=79.127.220.101:4041 , piece_id=MTYQKW3CLES2OMI5WQ3EBYUANNU7T73MPHR57FB7JNH6EBRGRODA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:27:26.626795101 (2h35m42.407728471s) [noise=true , address=156.146.43.225:4461 , piece_id=B5K343ACIE27RPLTJIVTVE3DWVMRWJEOHWFHBNXKN7A4J3RS2L7Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:1D4k7wnNtivVkHko5CSBuZH6zqMuFyfFfshkZwEPqnmpLads49, address:156.146.43.225:4461): protocol: storage node overloaded, request limit: 2000 Jul 27 00:27:27.001628703 (2h35m42.782562102s) [noise=true , address=79.127.220.97:4361 , piece_id=TXQYNILTXIKK4NAJ65EQ7JB45SS7PKYP7M5OZ5U2RYPVAWZ4476Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:27.061928944 (2h35m42.842862315s) [noise=true , address=79.127.205.235:4679 , piece_id=7BMKAKFYJSGBTDB64PQSNQLQCUDX64OOJJGAF3RBAYAERIZ3VNXA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:27.096086113 (2h35m42.877019476s) [noise=true , address=79.127.205.233:4642 , piece_id=WGKZBMVCFZH5SCCYCVDZYMBGLM2LW33QK45VYDCUXIYWMZUKVVHQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:27.703404942 (2h35m43.484338304s) [noise=true , address=79.127.205.245:4248 , piece_id=S25F3LV5UVV6UQRB27OSB76OAFVJJCQDC264D5SKIVFIH6B57KEQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:29.802564754 (2h35m45.583498125s) [noise=true , address=79.127.205.246:4278 , piece_id=JVCYSTU6YKUOPIWYBA2NTVNAI4TTT6N6JMLVCVTR6MM22TDZ5NTA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:33.581878489 (2h35m49.362811854s) [noise=true , address=64.49.13.139:4478 , piece_id=7OUHO2VKK4ZIDDQSBPDEG7KEBQP7GO6J2HNTF7N5Z3O22TTRUR3Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:28:26.387999475 (2h36m42.168932852s) [noise=true , address=104.36.82.56:4244 , piece_id=U2EIQHP5VR262NR53LOGXVQ7FOAJ5LYWDWFMB5YJRVIIXLXIKNMQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: write tcp 159.223.143.39:34230->104.36.82.56:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:26.388167306 (2h36m42.169100672s) [noise=true , address=77.90.31.17:4792 , piece_id=I2EFY5REBRIIWNQYGBT5PETJ4PVPUQPQFL27TG23PHOTNI7L4T4Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.388340549 (2h36m42.16927393s) [noise=true , address=104.36.82.47:4545 , piece_id=5HQAGGCDEHVM52TSNSX3O35ELQYNUO7DKHSJLWITZK4Q3KT7ZB6Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.388591403 (2h36m42.169524782s) [noise=true , address=104.36.82.54:4163 , piece_id=22FA4PAWJB4FQJ43NSJRN3K6UA6PCDK7CKFHPGGH6TDOFAF7YOXQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:26.388706433 (2h36m42.169639802s) [noise=true , address=104.36.82.52:4097 , piece_id=43MACCM5RLSY3IXHPFFYNAKJIQY45E6Z24N7TISGAEQSKHZPEVJA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM, address:104.36.82.52:4097): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.388810431 (2h36m42.169743798s) [noise=true , address=104.36.82.55:4202 , piece_id=NKVHLWTADQ6ALYP2Z5KBTRGY4YSC272YQXLOUTVWDRUFEYUCSKDA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.389041442 (2h36m42.169974821s) [noise=true , address=104.36.82.46:4507 , piece_id=SHIBRAZNU65Q5EXKZNJZZ6YT4OJPE2OZOQJH6MUD3EHBJEX4LACA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.389138404 (2h36m42.17007178s) [noise=true , address=104.36.82.11:4143 , piece_id=VWDZRME2V7OYFIK5EM7FPNHE2RQ4ALAZFEWOOWNTUZ5AGNFAPQMQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.389177161 (2h36m42.17011055s) [seg_pos={0 0}, upload=2392]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:28:26.424005881 (2h36m42.204939263s) [upload=2392]: Done uploading segments... Jul 27 00:28:26.424046045 (2h36m42.204979424s) [upload=2392]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.039877826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....X.X........X...XX...............................X......] duration: 42.550820116s FAIL (2025-07-27 00:28:24 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.550835699s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)