Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030509413s FAIL (2025-07-27 00:42:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== .19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169868067 (2h50m3.950801454s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.169883215 (2h50m3.950816602s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170323726 (2h50m3.951257115s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170477131 (2h50m3.951410511s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170776612 (2h50m3.951710001s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.170872777 (2h50m3.951806249s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171264996 (2h50m3.952198393s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171400116 (2h50m3.952333502s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.171677089 (2h50m3.952610465s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172136567 (2h50m3.953069943s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172340583 (2h50m3.953273992s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172501425 (2h50m3.953434806s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.172894778 (2h50m3.95382816s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173198440 (2h50m3.954131817s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.173488669 (2h50m3.954422039s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.176184947 (2h50m3.957118338s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Uploading piece... Jul 27 00:41:48.235658310 (2h50m4.016591695s) [upload=2618]: Next returned nil segment Jul 27 00:41:48.235681162 (2h50m4.016614548s) [upload=2618]: Waiting for error group managing segments... Jul 27 00:41:48.305694276 (2h50m4.086627643s) [noise=true , address=156.146.43.225:4448 , piece_id=GE2HYL44FCENZQQ223YKPVQNH26IW7T2V5YR5QTEAY3FQ6SWAJTA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:48.498540661 (2h50m4.279474038s) [noise=true , address=79.127.219.38:4017 , piece_id=4N4BDIZAPGTUTYWHOM5MJEBPXNMUEM3NTWKJ2QDJBYYR2IPYRA7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.726916798 (2h50m4.507850174s) [noise=true , address=79.127.205.246:4288 , piece_id=SZNQR2M6M5FPNNYEKBMRIY7WAUXXI34B3JLHRBZ6YJ7ZRFTQRRCA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.741500221 (2h50m4.522433583s) [noise=true , address=79.127.205.235:4687 , piece_id=M7WOFHGGGLCAITGJSQHW5YY7BY2GMMT7QCNT3XPPN4VLUC52ISHQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:48.781031242 (2h50m4.561964617s) [noise=true , address=45.91.116.10:4093 , piece_id=VQMCG65VUOYWPKXR7GAAGUTH5BSUC2GDD3K7J4ROLTDSQJXK3AOQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:41:54.101480874 (2h50m9.882414256s) [noise=true , address=45.91.116.14:4194 , piece_id=4YSYMV2PI2RKSDZIWOBEVWJBTR5E7AJRADYKIMH5YOQL3YXRDJ4Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:54.964393232 (2h50m10.745326661s) [noise=true , address=79.127.220.101:4041 , piece_id=557TCWKC7BUACZYZIIEFWVZBKWUBJLWTNCEAVRKE4SCJHDJHWR3A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:41:56.680213972 (2h50m12.461147333s) [noise=true , address=64.49.13.170:4717 , piece_id=PNL3T527ZRJPOU2UN5RTVBWJCJDITNKHGRHLW4HGVWSRTK3SWR2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:08.628924710 (2h50m24.409858089s) [noise=true , address=77.90.31.19:4844 , piece_id=WYWO6HNFXP7NXQZKXBIYSOOBPSONIR3JMOOFALYTYNBBCI4EP5GQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:22.681283266 (2h50m38.462216632s) [noise=true , address=77.90.31.13:4631 , piece_id=JJ2C5T5UWHW7WZXM3ALNZMKFVZDR4IX7DGCSK3QKR2KS3YVQAWOA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: <nil> Jul 27 00:42:48.107374941 (2h51m3.888308328s) [noise=true , address=104.36.82.44:4890 , piece_id=ZP33DLT5TD3HYC4QCZH73O4QMHBJJHEKD3X7QEGHJWSCGYIMKX7A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107451519 (2h51m3.8883849s) [noise=true , address=104.36.82.44:4902 , piece_id=5UPWC52CW544LARWS6DHXIPAKZHGP4E7OH53QYF63DY3VLRIXAGA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107460836 (2h51m3.888394218s) [noise=true , address=104.36.82.47:4545 , piece_id=GL2I7IVSKBHJTVGEHRO6KRNT3GRPNVDFVBHAGZH6O32DP77Z52MA, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107523416 (2h51m3.888456779s) [noise=true , address=104.36.82.54:4162 , piece_id=6LTGASCPGCOQQRS5MI6POGHM33YRJ7UXGLGUDBYAP7CDLWUQWL6A, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: write tcp 159.223.143.39:41074->104.36.82.54:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107540530 (2h51m3.888473913s) [noise=true , address=104.36.82.55:4216 , piece_id=LXNT7ITDGQWPR3YOGTOHG6PWEFQDS3SMROB2DOTDYWHPIKKOOLPQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107576870 (2h51m3.888510235s) [noise=true , address=104.36.82.46:4481 , piece_id=L6ZRTDJMGY5QFTR634QL4CU6QIIBZB6OH6L3TM6RE4L67JVYRGUQ, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107597455 (2h51m3.888530837s) [noise=true , address=104.36.82.53:4146 , piece_id=JMHHPXJKMPXK2MQXC4JBM7BN7RIHD6N3UECHBJVXPQXCD2PEV65Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:52750->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107634166 (2h51m3.888567547s) [noise=true , address=104.36.82.52:4087 , piece_id=WIFLTK5AMBFJMZIJV44SNPRMJNPD37D2JAEBFKZXWLCUEYEH3Q2Q, seg_pos={0 0}, upload=2618]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:42:48.107668612 (2h51m3.888601999s) [seg_pos={0 0}, upload=2618]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:42:48.135102625 (2h51m3.916036013s) [upload=2618]: Done uploading segments... Jul 27 00:42:48.135139340 (2h51m3.916072734s) [upload=2618]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.030553435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..XX...............................X...............X...X....] duration: 4.304492884s FAIL (2025-07-27 00:42:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.304557323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...XX...............................X...............X...X...] duration: 45.657032924s FAIL (2025-07-27 00:42:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.65706318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XX....X...X.................XX....X......XX.XXXX...X......] duration: 589.80399ms FAIL (2025-07-27 00:42:22 UTC) orbiter warning: context deadline exceeded Duration: 589.877731ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX..] duration: 1.923125569s FAIL (2025-07-27 00:42:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.923156928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 223.928231ms FAIL (2025-07-27 00:42:08 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 223.97213ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.XX....X...X.................XX....X......XX.XXXX...X.....] duration: 1.863228781s FAIL (2025-07-27 00:42:08 UTC) orbiter warning: context deadline exceeded Duration: 1.863317329s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X...XX...............................X...............X...X..] duration: 34.039026214s FAIL (2025-07-27 00:41:47 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.039053715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.032012554s FAIL (2025-07-27 00:41:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== h49m2.725838469s) [upload=2603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:46.944916861 (2h49m2.72585025s) [upload=2603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:46.993610734 (2h49m2.774544131s) [seg_pos={0 0}, upload=2603]: Begin upload segment... Jul 27 00:40:46.993662784 (2h49m2.774596167s) [seg_pos={0 0}, upload=2603]: Done begin upload segment. Jul 27 00:40:46.993701077 (2h49m2.774634457s) [noise=true , address=79.127.201.215:4359 , piece_id=CLKFAOV2YFGMRSLI6P7WBOVR2FP5EY7VG5IVBMBHXUUC5V77PAVQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.993725626 (2h49m2.774659015s) [noise=true , address=79.127.205.246:4278 , piece_id=ZLAWGCCYP7K4WPPXRSMF2UICZHJ4TKHZHZKFHQFRZCWWUHWPNUHA, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.994337899 (2h49m2.775271289s) [noise=true , address=64.49.13.138:4483 , piece_id=FLYFNXGQAZ7XINSFJILIMYS7THRJBPZ2TASUVNWXBEKDZZTX6U6A, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.994539036 (2h49m2.775472426s) [noise=true , address=79.127.205.235:4677 , piece_id=4M7A7FDFLOFKEDXYEZIW4LMSKC7VEQB3L5ZEQVCX5EP36NB7L5UQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.994724098 (2h49m2.77565748s) [noise=true , address=77.90.31.17:4793 , piece_id=TO47NJLJAXTRHKF3HRKCKHJWJMV45JP6HJEHV3HV5ZDAUYBYUD5A, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.995123442 (2h49m2.776056828s) [noise=true , address=104.36.82.47:4522 , piece_id=HTUZ5A3GMYJB5ZZ24ODI5UNHKXHEPSYR7GIUDMHB7UJDEGZOMJUQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.995237580 (2h49m2.776170964s) [noise=true , address=77.90.31.10:4533 , piece_id=7ETUKWH3JR5GE2W6D7CWWY6WOY4DWDQR7Q6CIEHGWQCXMS2LFXPQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.995624425 (2h49m2.776557809s) [noise=true , address=45.91.116.10:4093 , piece_id=E3DP5WQGR4NHAFENWEKOHRLFKJ7D5OKUO4ZJ7ROMAOVVDI3VBT2A, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.996248130 (2h49m2.777181516s) [noise=true , address=79.127.205.234:4603 , piece_id=S3HJ4LWLB67TO44Q3CN3ASCC6W3CFLRWGI5PIEW2TLWHRQO23W7A, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.996573244 (2h49m2.777506632s) [noise=true , address=45.91.116.9:4277 , piece_id=KMJHKWLHZKF3SVCUMGPIPTHTUHMMY7UYUY3ZR2TVFAREQX7LTVEA, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.996967974 (2h49m2.777901362s) [noise=true , address=45.91.116.14:4194 , piece_id=VRKDVE4UKOU5SW2YLN2YTXG3RYX4L3KOBYQTD7JPVFWVLJHJW7QQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.997579429 (2h49m2.778512816s) [noise=true , address=104.36.82.55:4197 , piece_id=OO7XNGA2742DW4LYYXEITQJ5HFBCPO7WODDJ6AVACVWKWBUHM7IA, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.997948809 (2h49m2.778882185s) [noise=true , address=77.90.31.19:4847 , piece_id=7XYEDY6RE3ACSF44OYBY25G2HFVKVS6UBV3APVGTU6MJ6ICTZJUQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.998255255 (2h49m2.779188635s) [noise=true , address=104.36.82.52:4115 , piece_id=JALXRCMGLRYAWLNV3CMZSXCQU4IG5VYCFZFEKEM67HCDYTESGLNQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.998546646 (2h49m2.779480029s) [noise=true , address=104.36.82.46:4485 , piece_id=WOPT53YODU7PP72D76OI4FO3MBWF4TFPNZ4YY73OGVGCT3SNKOSQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:46.999627355 (2h49m2.780560738s) [noise=true , address=79.127.205.245:4239 , piece_id=EP2HWX5T64WHM44DG3ECIIIY4H2DJF4BSS4VONP4OTOIKNCNZLYQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:47.000332427 (2h49m2.78126581s) [noise=true , address=79.127.219.47:4048 , piece_id=ZXZJC5VJSY2HGTJGVGHZC7M52XAL6C7HMRXAMV3VWQ7JPT43TDTQ, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:47.000672690 (2h49m2.781606059s) [noise=true , address=77.90.31.14:4683 , piece_id=KHK63ZDU5655CDV7LS3WSDYPQCK2KW7NXU3EV6YJ6T4MIRWV64IA, seg_pos={0 0}, upload=2603]: Uploading piece... Jul 27 00:40:47.101108121 (2h49m2.88204151s) [upload=2603]: Next returned nil segment Jul 27 00:40:47.101133388 (2h49m2.882066786s) [upload=2603]: Waiting for error group managing segments... Jul 27 00:40:47.187964419 (2h49m2.968897794s) [noise=true , address=77.90.31.14:4683 , piece_id=KHK63ZDU5655CDV7LS3WSDYPQCK2KW7NXU3EV6YJ6T4MIRWV64IA, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:40:47.640125863 (2h49m3.421059254s) [noise=true , address=79.127.205.234:4603 , piece_id=S3HJ4LWLB67TO44Q3CN3ASCC6W3CFLRWGI5PIEW2TLWHRQO23W7A, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:40:47.644585196 (2h49m3.425518567s) [noise=true , address=79.127.205.235:4677 , piece_id=4M7A7FDFLOFKEDXYEZIW4LMSKC7VEQB3L5ZEQVCX5EP36NB7L5UQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:40:47.664709414 (2h49m3.445642794s) [noise=true , address=79.127.205.246:4278 , piece_id=ZLAWGCCYP7K4WPPXRSMF2UICZHJ4TKHZHZKFHQFRZCWWUHWPNUHA, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:40:47.772213845 (2h49m3.553147224s) [noise=true , address=79.127.205.245:4239 , piece_id=EP2HWX5T64WHM44DG3ECIIIY4H2DJF4BSS4VONP4OTOIKNCNZLYQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:40:48.637647173 (2h49m4.418580549s) [noise=true , address=45.91.116.14:4194 , piece_id=VRKDVE4UKOU5SW2YLN2YTXG3RYX4L3KOBYQTD7JPVFWVLJHJW7QQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 27 00:40:49.032537186 (2h49m4.813470564s) [noise=true , address=45.91.116.9:4277 , piece_id=KMJHKWLHZKF3SVCUMGPIPTHTUHMMY7UYUY3ZR2TVFAREQX7LTVEA, seg_pos={0 0}, upload=2603]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:40:50.411906732 (2h49m6.192840109s) [noise=true , address=79.127.219.47:4048 , piece_id=ZXZJC5VJSY2HGTJGVGHZC7M52XAL6C7HMRXAMV3VWQ7JPT43TDTQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:40:50.510389420 (2h49m6.291322805s) [noise=true , address=64.49.13.138:4483 , piece_id=FLYFNXGQAZ7XINSFJILIMYS7THRJBPZ2TASUVNWXBEKDZZTX6U6A, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:41:01.587368630 (2h49m17.368302001s) [noise=true , address=45.91.116.10:4093 , piece_id=E3DP5WQGR4NHAFENWEKOHRLFKJ7D5OKUO4ZJ7ROMAOVVDI3VBT2A, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:41:01.613191588 (2h49m17.394124971s) [noise=true , address=77.90.31.19:4847 , piece_id=7XYEDY6RE3ACSF44OYBY25G2HFVKVS6UBV3APVGTU6MJ6ICTZJUQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:41:03.770516266 (2h49m19.551449634s) [noise=true , address=79.127.201.215:4359 , piece_id=CLKFAOV2YFGMRSLI6P7WBOVR2FP5EY7VG5IVBMBHXUUC5V77PAVQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:41:14.400180452 (2h49m30.181113822s) [noise=true , address=77.90.31.10:4533 , piece_id=7ETUKWH3JR5GE2W6D7CWWY6WOY4DWDQR7Q6CIEHGWQCXMS2LFXPQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: <nil> Jul 27 00:41:46.902476399 (2h50m2.683409768s) [noise=true , address=104.36.82.47:4522 , piece_id=HTUZ5A3GMYJB5ZZ24ODI5UNHKXHEPSYR7GIUDMHB7UJDEGZOMJUQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: write tcp 159.223.143.39:39966->104.36.82.47:4522: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:46.902648432 (2h50m2.683581821s) [noise=true , address=77.90.31.17:4793 , piece_id=TO47NJLJAXTRHKF3HRKCKHJWJMV45JP6HJEHV3HV5ZDAUYBYUD5A, seg_pos={0 0}, upload=2603]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:46.902733784 (2h50m2.68366716s) [noise=true , address=104.36.82.52:4115 , piece_id=JALXRCMGLRYAWLNV3CMZSXCQU4IG5VYCFZFEKEM67HCDYTESGLNQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:46.902761735 (2h50m2.683695114s) [noise=true , address=104.36.82.46:4485 , piece_id=WOPT53YODU7PP72D76OI4FO3MBWF4TFPNZ4YY73OGVGCT3SNKOSQ, seg_pos={0 0}, upload=2603]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: rpc: dial tcp 104.36.82.46:4485: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:46.902819011 (2h50m2.683752378s) [noise=true , address=104.36.82.55:4197 , piece_id=OO7XNGA2742DW4LYYXEITQJ5HFBCPO7WODDJ6AVACVWKWBUHM7IA, seg_pos={0 0}, upload=2603]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: write tcp 159.223.143.39:40824->104.36.82.55:4197: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:46.902877363 (2h50m2.683810743s) [seg_pos={0 0}, upload=2603]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:41:46.932083949 (2h50m2.71301733s) [upload=2603]: Done uploading segments... Jul 27 00:41:46.932127819 (2h50m2.713061197s) [upload=2603]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.032034791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [X...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX.] duration: 1.944674041s FAIL (2025-07-27 00:41:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.944712504s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.XX....X...X.................XX....X......XX.XXXX...X....] duration: 576.93913ms FAIL (2025-07-27 00:41:21 UTC) orbiter warning: context deadline exceeded Duration: 576.970331ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 201.884082ms FAIL (2025-07-27 00:41:17 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 201.932355ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [XX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...XX] duration: 30.037446143s FAIL (2025-07-27 00:41:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.037488378s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X...XX...............................X...............X...X.] duration: 15.705042347s FAIL (2025-07-27 00:41:04 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.705060141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX.] duration: 642.257324ms FAIL (2025-07-27 00:41:03 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 642.287172ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.XX] duration: 30.037127365s FAIL (2025-07-27 00:41:02 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Bh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:40:32.301975791 (2h48m48.082909176s) [upload=2598]: Starting upload Jul 27 00:40:32.302058440 (2h48m48.082991829s) [upload=2598]: Done starting upload Jul 27 00:40:32.302435745 (2h48m48.083369144s) [upload=2598]: Uploading segments... Jul 27 00:40:32.302495145 (2h48m48.08342853s) [upload=2598]: Got next segment. Inline: false Jul 27 00:40:32.302534583 (2h48m48.083467968s) [upload=2598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:32.302542581 (2h48m48.083475965s) [upload=2598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:32.328630022 (2h48m48.109563428s) [seg_pos={0 0}, upload=2598]: Begin upload segment... Jul 27 00:40:32.328689143 (2h48m48.109622517s) [seg_pos={0 0}, upload=2598]: Done begin upload segment. Jul 27 00:40:32.328722109 (2h48m48.109655504s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329277137 (2h48m48.110210531s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.329702178 (2h48m48.110635562s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330219571 (2h48m48.11115296s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330608177 (2h48m48.111541565s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.330917689 (2h48m48.111851076s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.331375066 (2h48m48.112308453s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.332673152 (2h48m48.113606577s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.333197874 (2h48m48.114131261s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.334525211 (2h48m48.115458599s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335328252 (2h48m48.116261629s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.335722309 (2h48m48.116655717s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.336199174 (2h48m48.117132555s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.337793830 (2h48m48.118727213s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338560128 (2h48m48.119493505s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.338908792 (2h48m48.119842175s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Uploading piece... Jul 27 00:40:32.612474740 (2h48m48.39340812s) [upload=2598]: Next returned nil segment Jul 27 00:40:32.612496838 (2h48m48.393430234s) [upload=2598]: Waiting for error group managing segments... Jul 27 00:40:32.619754745 (2h48m48.400688141s) [noise=true , address=45.91.116.6:4975 , piece_id=POGES2S7XE7NWAPXQIZJD5JX7ZS4X3S2ODCT4TJD2B4XMRW7RZUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.620609662 (2h48m48.40154304s) [noise=true , address=45.91.116.10:4083 , piece_id=UR2ZYLCHGXOQBYC5GB62KHVKIZLIIFRFS2QA3ZMIL4VUZLKWAVEA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.624474798 (2h48m48.405408163s) [noise=true , address=45.91.116.9:4261 , piece_id=S6ZA2FFMQFG26JJQRAGYF7MSKATLJLNRRIYEVBIQXAFNQUET3W5Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.626118614 (2h48m48.407051982s) [noise=true , address=45.91.116.5:4298 , piece_id=MRK5EUL2KXXSZSVYBMN7O4AVI7MSXYCCUD7KLQZFYKYOCSVBMGUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627057615 (2h48m48.407990996s) [noise=true , address=45.91.116.4:4939 , piece_id=M3MGTURRVGWSEJVPWTXRX6VIHZQLKL2YFK64XIMZ6IKMFDMJTAGA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.627959898 (2h48m48.408893271s) [noise=true , address=45.91.116.11:4117 , piece_id=5BSNWDYJ6WAOTTTBPALHQ7PV4IXE3IPPTX6O7XKVKPULUESPRNIQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.628862646 (2h48m48.409796018s) [noise=true , address=45.91.116.8:4048 , piece_id=V5PWKJW3SZKTT67U7VECHTSP5APK2NNQGTQX4H2CYYUXDMRUP32Q, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.629913398 (2h48m48.410846763s) [noise=true , address=77.90.31.16:4735 , piece_id=JXGB6PVLHFYPWQCEJUISIJPYVSVXKA7Q2K2SFL3NADVF4PSAK5NA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.630734494 (2h48m48.411667867s) [noise=true , address=45.91.116.7:4009 , piece_id=LVUADZYDUI52TKW46HTKOUAQVUP4JVPYSIRTFJETTV7R5YYNCPUA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.633794198 (2h48m48.414727576s) [noise=true , address=45.91.116.10:4082 , piece_id=MRWUFLQK7IVPGXHFHQLDWRZXA5Y4E3JMIHEH6EGECUYWPUGFX43A, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:32.783415593 (2h48m48.564348964s) [noise=true , address=45.91.116.14:4191 , piece_id=3IMBRJSYRILH6EJI7EWIAAS4J7USB5CGN4FWNJYJXFFDYDGU3XKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:40:34.005531433 (2h48m49.786464815s) [noise=true , address=77.90.31.13:4628 , piece_id=UBFSF5TGWIL6A2M472HLJ65GBNAMN2EKNSDJGHK2DFUDYXULYTHQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: <nil> Jul 27 00:41:02.264893026 (2h49m18.045826412s) [noise=true , address=104.36.82.10:4079 , piece_id=E4HPQDGRPELTNLH3SCBKKTDCTE4LBWPPI5DEHWB42S26NWPHCYYA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58124->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.264991804 (2h49m18.04592519s) [noise=true , address=104.36.82.5:4839 , piece_id=LZDLKKEGH6BMU6YJ22JNF6EMQN2QUYF4RCXK3QD4XXMI6BGMGGZQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:60974->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272010916 (2h49m18.052944316s) [noise=true , address=104.36.82.10:4077 , piece_id=JO2GYIKRKARWXPDMOMB543Z3ETVNBCZ6IVDK5TCFVBKQCAWV72XQ, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:56424->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:02.272204522 (2h49m18.053137902s) [noise=true , address=104.36.82.15:4259 , piece_id=54UUM6WMJPWIGMFINZIDPC2ZHTG3463IKWO35XL7XRDZOQJ7EEKA, seg_pos={0 0}, upload=2598]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:41:02.272255696 (2h49m18.05318908s) [seg_pos={0 0}, upload=2598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:41:02.297648019 (2h49m18.078581393s) [upload=2598]: Done uploading segments... Jul 27 00:41:02.297678403 (2h49m18.078611773s) [upload=2598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.037149165s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X.XX....X...X.................XX....X......XX.XXXX...X...] duration: 1.761217988s FAIL (2025-07-27 00:40:52 UTC) orbiter warning: context deadline exceeded Duration: 1.761250124s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X...XX...............................X...............X...X] duration: 1m0.000780903s FAIL (2025-07-27 00:40:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00080881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 456.54473ms FAIL (2025-07-27 00:40:48 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 456.576373ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.02779495s FAIL (2025-07-27 00:40:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== e_id=A65MGIK4DYCHK2T7XFRE35GMYFWEM45MJIEEG5W5Y7BHAWYJ7H6Q, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.880932841 (2h48m2.661866219s) [noise=true , address=104.36.82.47:4533 , piece_id=IGDCLPZGPAMASBY4KOE5BTN6N736RDOYWQVYS3S3XJ2OQCT3ZZ6A, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.881205690 (2h48m2.662139071s) [noise=true , address=104.36.82.39:4732 , piece_id=FIQZZULKSGQYM3AA45SKCPEAOWR3JIPCXNUZXFQJSWCHTSLWFLGQ, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.881532467 (2h48m2.662465842s) [noise=true , address=77.90.31.10:4533 , piece_id=SEQYYUV6BPNKS7PS64I3CCFPIQMPMJPYUOY533RNS5BUXCTACLYA, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.881852975 (2h48m2.662786592s) [noise=true , address=104.36.82.6:4903 , piece_id=QSX3FJGAI7U26JAA2IFW5EJFDU6257DKGIEOMXHR2P22BHCO2HOQ, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.882112398 (2h48m2.663045776s) [noise=true , address=104.36.82.46:4484 , piece_id=7YUGHBQNQ7QHFQGSFMWOLM6FPNRO4XKEU5SYFW5AEDUNHNZXZLHA, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.882343990 (2h48m2.663277365s) [noise=true , address=79.127.205.245:4221 , piece_id=FZ6E5DPTNFVJMMZDZFTSZED74RO7BEOURSWVTU7EXPAEBUSVGNNA, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.882746464 (2h48m2.663679849s) [noise=true , address=104.36.82.15:4271 , piece_id=55MOFUXVMJDXFBDIQTCC7J5JCRUJUQZLXYHOYIQILKDDZFH2EXKQ, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.883129953 (2h48m2.664063339s) [noise=true , address=45.91.116.14:4194 , piece_id=HJHTWGKEHDZXKIDGMZHHZLO6P3R63A7I3INSU33SID6YG33HSPDQ, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.883794042 (2h48m2.664727428s) [noise=true , address=104.36.82.52:4102 , piece_id=H4UMKCKAFPSJSAYPCUKLCW6OYPV44SILOB4LYJYLVZ6FD4X7JBLA, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.884141825 (2h48m2.665075204s) [noise=true , address=79.127.219.38:4015 , piece_id=JQ7C3HWUH3666GMERVVNSSLZUCYWW7K45DMFNQI5Y65CJFIDR6QQ, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.884428437 (2h48m2.665361812s) [noise=true , address=79.127.205.246:4283 , piece_id=FDZ2HHVP73K4P43ZED2OTNCXE32A6K3H5E6LAESKSSXIMIFDSZBA, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.884735035 (2h48m2.665668417s) [noise=true , address=104.36.82.54:4182 , piece_id=GTVJSDQ2IBKQACKUGZGBBBCLKJ4WY66JPUEMMT2JA2CV4LKO3WXQ, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.885122377 (2h48m2.666055756s) [noise=true , address=104.36.82.56:4233 , piece_id=OHYJSMT7G2JNIINV7ASXKXM5CIOE37YSRWTP3GNPRSPCJG4ZXVFA, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.885461694 (2h48m2.666395058s) [noise=true , address=77.90.31.11:4581 , piece_id=Y5YBIWGEANBXDO7LBEKTWARY5UGBVQLDZLTC2AZWHFKGKTRS2J6A, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.885779471 (2h48m2.66671285s) [noise=true , address=45.91.116.9:4277 , piece_id=WCXZQDTIGWZSIHJUAXZTNRQ2QXWSQ3SGKH3SXZ3RVRPP6BGRYNTQ, seg_pos={0 0}, upload=2586]: Uploading piece... Jul 27 00:39:46.974794394 (2h48m2.755727771s) [upload=2586]: Next returned nil segment Jul 27 00:39:46.974814383 (2h48m2.755747797s) [upload=2586]: Waiting for error group managing segments... Jul 27 00:39:47.020749187 (2h48m2.801682565s) [noise=true , address=77.90.31.11:4581 , piece_id=Y5YBIWGEANBXDO7LBEKTWARY5UGBVQLDZLTC2AZWHFKGKTRS2J6A, seg_pos={0 0}, upload=2586]: Done uploading piece. err: <nil> Jul 27 00:39:47.026346401 (2h48m2.807279782s) [noise=true , address=45.91.116.10:4093 , piece_id=IZ3UF7LPYYYMLTIJ7GVTDR7IIGK76GPZFHNATPMUJHCCAWZY5XMA, seg_pos={0 0}, upload=2586]: Done uploading piece. err: <nil> Jul 27 00:39:47.091636225 (2h48m2.872569604s) [noise=true , address=79.127.219.38:4015 , piece_id=JQ7C3HWUH3666GMERVVNSSLZUCYWW7K45DMFNQI5Y65CJFIDR6QQ, seg_pos={0 0}, upload=2586]: Done uploading piece. err: <nil> Jul 27 00:39:47.523282480 (2h48m3.304215843s) [noise=true , address=79.127.205.245:4221 , piece_id=FZ6E5DPTNFVJMMZDZFTSZED74RO7BEOURSWVTU7EXPAEBUSVGNNA, seg_pos={0 0}, upload=2586]: Done uploading piece. err: <nil> Jul 27 00:39:47.539256701 (2h48m3.320190067s) [noise=true , address=79.127.205.246:4283 , piece_id=FDZ2HHVP73K4P43ZED2OTNCXE32A6K3H5E6LAESKSSXIMIFDSZBA, seg_pos={0 0}, upload=2586]: Done uploading piece. err: <nil> Jul 27 00:39:47.994396611 (2h48m3.775329995s) [noise=true , address=45.91.116.14:4194 , piece_id=HJHTWGKEHDZXKIDGMZHHZLO6P3R63A7I3INSU33SID6YG33HSPDQ, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 27 00:39:50.134125957 (2h48m5.915059331s) [noise=true , address=79.127.201.214:4321 , piece_id=BXH52TMFXONCONT47MIOQZYDAH52SNGCBSQI33VZ44G7V35CR4QA, seg_pos={0 0}, upload=2586]: Done uploading piece. err: <nil> Jul 27 00:40:12.188871474 (2h48m27.969804837s) [noise=true , address=45.91.116.9:4277 , piece_id=WCXZQDTIGWZSIHJUAXZTNRQ2QXWSQ3SGKH3SXZ3RVRPP6BGRYNTQ, seg_pos={0 0}, upload=2586]: Done uploading piece. err: <nil> Jul 27 00:40:46.808057883 (2h49m2.588991266s) [noise=true , address=77.90.31.10:4533 , piece_id=SEQYYUV6BPNKS7PS64I3CCFPIQMPMJPYUOY533RNS5BUXCTACLYA, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:40:46.808309046 (2h49m2.589242421s) [noise=true , address=104.36.82.56:4233 , piece_id=OHYJSMT7G2JNIINV7ASXKXM5CIOE37YSRWTP3GNPRSPCJG4ZXVFA, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808438463 (2h49m2.589371846s) [noise=true , address=104.36.82.47:4533 , piece_id=IGDCLPZGPAMASBY4KOE5BTN6N736RDOYWQVYS3S3XJ2OQCT3ZZ6A, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: rpc: dial tcp 104.36.82.47:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808485884 (2h49m2.589419263s) [noise=true , address=104.36.82.52:4102 , piece_id=H4UMKCKAFPSJSAYPCUKLCW6OYPV44SILOB4LYJYLVZ6FD4X7JBLA, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: write tcp 159.223.143.39:44628->104.36.82.52:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808575033 (2h49m2.589508409s) [noise=true , address=104.36.82.39:4732 , piece_id=FIQZZULKSGQYM3AA45SKCPEAOWR3JIPCXNUZXFQJSWCHTSLWFLGQ, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:171oGgyDi35YSen7MKLmHUeQjZKwNAoYt5AX4D34YrHaBEoiLH, address:104.36.82.39:4732): protocol: rpc: dial tcp 104.36.82.39:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808596964 (2h49m2.589530346s) [noise=true , address=104.36.82.6:4903 , piece_id=QSX3FJGAI7U26JAA2IFW5EJFDU6257DKGIEOMXHR2P22BHCO2HOQ, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:1zFBuDaxAtbFho2wumoXWSoQbB7y9VaSFizz31gV8n1fLK6syE, address:104.36.82.6:4903): protocol: rpc: dial tcp 104.36.82.6:4903: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808641969 (2h49m2.589575345s) [noise=true , address=104.36.82.54:4182 , piece_id=GTVJSDQ2IBKQACKUGZGBBBCLKJ4WY66JPUEMMT2JA2CV4LKO3WXQ, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:12HbkH31sfDUnZLXAbmzQphcFYmzB8MQo98Q6Ufbo59et42zxMy, address:104.36.82.54:4182): protocol: write tcp 159.223.143.39:46000->104.36.82.54:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808708553 (2h49m2.58964193s) [noise=true , address=104.36.82.46:4484 , piece_id=7YUGHBQNQ7QHFQGSFMWOLM6FPNRO4XKEU5SYFW5AEDUNHNZXZLHA, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: rpc: dial tcp 104.36.82.46:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808739740 (2h49m2.589673101s) [noise=true , address=104.36.82.15:4271 , piece_id=55MOFUXVMJDXFBDIQTCC7J5JCRUJUQZLXYHOYIQILKDDZFH2EXKQ, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu, address:104.36.82.15:4271): protocol: rpc: dial tcp 104.36.82.15:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808780391 (2h49m2.589713766s) [noise=true , address=104.36.82.53:4147 , piece_id=A65MGIK4DYCHK2T7XFRE35GMYFWEM45MJIEEG5W5Y7BHAWYJ7H6Q, seg_pos={0 0}, upload=2586]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:46.808815496 (2h49m2.589748883s) [seg_pos={0 0}, upload=2586]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:40:46.833437623 (2h49m2.614370989s) [upload=2586]: Done uploading segments... Jul 27 00:40:46.833484447 (2h49m2.614417834s) [upload=2586]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.027819475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...X] duration: 30.031209998s FAIL (2025-07-27 00:40:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.031340477s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.X] duration: 30.034940216s FAIL (2025-07-27 00:40:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h47m49.180818075s) [noise=true , address=104.36.82.16:4293 , piece_id=GE7A7H5OXGCHSWCEXCOIZJ4K2UC5YMN5EZ7SEUVFBHCOUKO3347Q, seg_pos={0 0}, upload=2582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 159.223.143.39:59522->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:33.399905504 (2h47m49.18083888s) [seg_pos={0 0}, upload=2582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:39:33.400023153 (2h47m49.18095653s) [upload=2582]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:39:33.400030894 (2h47m49.180964278s) [upload=2582]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:39:33.456823251 (2h47m49.237756614s) [upload=2582]: Done uploading segments... Jul 27 00:39:33.456843817 (2h47m49.23777718s) [upload=2582]: Upload finished. err: <nil> Jul 27 00:40:02.204546852 (2h48m17.985480239s) [upload=2588]: Starting upload Jul 27 00:40:02.204629253 (2h48m17.985562638s) [upload=2588]: Done starting upload Jul 27 00:40:02.205729153 (2h48m17.986662527s) [upload=2588]: Uploading segments... Jul 27 00:40:02.205843999 (2h48m17.986777389s) [upload=2588]: Got next segment. Inline: false Jul 27 00:40:02.206102925 (2h48m17.987036308s) [upload=2588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:40:02.206113402 (2h48m17.987046792s) [upload=2588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:40:02.251488581 (2h48m18.032421964s) [seg_pos={0 0}, upload=2588]: Begin upload segment... Jul 27 00:40:02.251552558 (2h48m18.032485948s) [seg_pos={0 0}, upload=2588]: Done begin upload segment. Jul 27 00:40:02.251591991 (2h48m18.032525381s) [noise=true , address=45.91.116.4:4939 , piece_id=SUILQIZKVZQNNQJUWKGN2C5OYZP6T6T4DMDWOZZSSELDTROKG2QQ, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.252132745 (2h48m18.033066134s) [noise=true , address=77.90.31.14:4661 , piece_id=2PZNX3KQXAUVJL4NB27JUUZ3UGGF5LTKDF5MBN3EQJKAOHKFWS7Q, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.252534245 (2h48m18.033467635s) [noise=true , address=45.91.116.6:4973 , piece_id=NWF5T74XPTIURIF7JX2Y53HEZFFW67PRHJ53EYGLPCIN6AXAYX4Q, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.252938253 (2h48m18.033871641s) [noise=true , address=104.36.82.2:4731 , piece_id=UL7R2WITGLTNH57CLJVZGD3ABH6ZZUDOEXL62Y7WMNF5CTHA722A, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.253352806 (2h48m18.034286194s) [noise=true , address=77.90.31.16:4733 , piece_id=RJH3LQLABXHL574VMZQOVKVSRXKXKCW2RCCTRFOQU65DVUHXB6SA, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.253730750 (2h48m18.034664135s) [noise=true , address=104.36.82.4:4804 , piece_id=QVECRXEWXS6URT74UD5TJOF7HV76SH3Z7AALJBVZP2XUIXY6HSOQ, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.254120709 (2h48m18.035054098s) [noise=true , address=45.91.116.7:4009 , piece_id=G5ROI6OOT5MDTPS6WYXUNTBTQBTOAOSXUAZHUNYIDRNX4XE6TXKQ, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.254490510 (2h48m18.035423896s) [noise=true , address=45.91.116.13:4155 , piece_id=6DEDC2QJJRJLQRX26SW7QEW3PNLUZJWZCQT5CGO6TCXHUM45MAEA, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.258595492 (2h48m18.039528882s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.259197974 (2h48m18.040131363s) [noise=true , address=104.36.82.7:4910 , piece_id=YSX2GCPQITG653ZFAAMGK62HBKEZZMVYSVP4WALNSBCXYLBHQMVA, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.259638226 (2h48m18.040571615s) [noise=true , address=77.90.31.12:4592 , piece_id=RC5PBNVNJINJLT3WMMUJYV4S445K2QUS46EVJBMCK3LN7AIRIB3Q, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.260061800 (2h48m18.040995189s) [noise=true , address=45.91.116.11:4118 , piece_id=CICRJ73AOY5755EBLLL5MYNQ5JEH5QCVXW7JFJN4D7YIXGYCLGEQ, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.260447699 (2h48m18.041381088s) [noise=true , address=77.90.31.11:4553 , piece_id=4QE5W74JLVZLNNS5MRPAPMFF3ZBEX5TRYXZYE33RRYLSWH2JLYMQ, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.260803606 (2h48m18.041736992s) [noise=true , address=45.91.116.14:4189 , piece_id=UZA4COSMGIRV4BVDYEA5LKCYVW6FZFRMN7QMBCB3PY7N6LEC4D7Q, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.261213954 (2h48m18.042147345s) [noise=true , address=77.90.31.19:4843 , piece_id=HWABBZAVF2X32V32H6ELI7ODKZO4SCBNARYDETXTSDPBB7TDWBEA, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.262281834 (2h48m18.04321522s) [noise=true , address=45.91.116.10:4081 , piece_id=3OOXY2CI2EFEHTKZVUSJ5I7DWD7H5QOQZIXJKCKCS2PHEUUHMHMQ, seg_pos={0 0}, upload=2588]: Uploading piece... Jul 27 00:40:02.429657485 (2h48m18.210590866s) [upload=2588]: Next returned nil segment Jul 27 00:40:02.432254895 (2h48m18.213188285s) [upload=2588]: Waiting for error group managing segments... Jul 27 00:40:02.441272449 (2h48m18.222205829s) [noise=true , address=77.90.31.11:4553 , piece_id=4QE5W74JLVZLNNS5MRPAPMFF3ZBEX5TRYXZYE33RRYLSWH2JLYMQ, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.442724448 (2h48m18.223657828s) [noise=true , address=45.91.116.11:4118 , piece_id=CICRJ73AOY5755EBLLL5MYNQ5JEH5QCVXW7JFJN4D7YIXGYCLGEQ, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.443418138 (2h48m18.224351541s) [noise=true , address=77.90.31.16:4733 , piece_id=RJH3LQLABXHL574VMZQOVKVSRXKXKCW2RCCTRFOQU65DVUHXB6SA, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.443832343 (2h48m18.224765714s) [noise=true , address=45.91.116.13:4155 , piece_id=6DEDC2QJJRJLQRX26SW7QEW3PNLUZJWZCQT5CGO6TCXHUM45MAEA, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.450010554 (2h48m18.230943923s) [noise=true , address=45.91.116.6:4973 , piece_id=NWF5T74XPTIURIF7JX2Y53HEZFFW67PRHJ53EYGLPCIN6AXAYX4Q, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.450463123 (2h48m18.231396485s) [noise=true , address=45.91.116.7:4009 , piece_id=G5ROI6OOT5MDTPS6WYXUNTBTQBTOAOSXUAZHUNYIDRNX4XE6TXKQ, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.451153475 (2h48m18.232086853s) [noise=true , address=45.91.116.10:4081 , piece_id=3OOXY2CI2EFEHTKZVUSJ5I7DWD7H5QOQZIXJKCKCS2PHEUUHMHMQ, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.451927990 (2h48m18.232861353s) [noise=true , address=45.91.116.4:4939 , piece_id=SUILQIZKVZQNNQJUWKGN2C5OYZP6T6T4DMDWOZZSSELDTROKG2QQ, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.452855642 (2h48m18.233789016s) [noise=true , address=77.90.31.14:4661 , piece_id=2PZNX3KQXAUVJL4NB27JUUZ3UGGF5LTKDF5MBN3EQJKAOHKFWS7Q, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:02.525718036 (2h48m18.306651415s) [noise=true , address=45.91.116.14:4189 , piece_id=UZA4COSMGIRV4BVDYEA5LKCYVW6FZFRMN7QMBCB3PY7N6LEC4D7Q, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:03.838040163 (2h48m19.618973531s) [noise=true , address=77.90.31.19:4843 , piece_id=HWABBZAVF2X32V32H6ELI7ODKZO4SCBNARYDETXTSDPBB7TDWBEA, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:04.618625381 (2h48m20.399558765s) [noise=true , address=77.90.31.12:4592 , piece_id=RC5PBNVNJINJLT3WMMUJYV4S445K2QUS46EVJBMCK3LN7AIRIB3Q, seg_pos={0 0}, upload=2588]: Done uploading piece. err: <nil> Jul 27 00:40:32.163269613 (2h48m47.944202991s) [noise=true , address=104.36.82.2:4731 , piece_id=UL7R2WITGLTNH57CLJVZGD3ABH6ZZUDOEXL62Y7WMNF5CTHA722A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163460916 (2h48m47.944394295s) [noise=true , address=104.36.82.4:4804 , piece_id=QVECRXEWXS6URT74UD5TJOF7HV76SH3Z7AALJBVZP2XUIXY6HSOQ, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:59332->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163564121 (2h48m47.944497491s) [noise=true , address=104.36.82.7:4910 , piece_id=YSX2GCPQITG653ZFAAMGK62HBKEZZMVYSVP4WALNSBCXYLBHQMVA, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163652770 (2h48m47.944586149s) [noise=true , address=104.36.82.10:4079 , piece_id=4OBIE4FORG4FWEO2MJAFBCYOQ2PGTFJY555GMBEOXVIMITEPO47A, seg_pos={0 0}, upload=2588]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:40:32.163844561 (2h48m47.944777946s) [seg_pos={0 0}, upload=2588]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:40:32.196349503 (2h48m47.977282886s) [upload=2588]: Done uploading segments... Jul 27 00:40:32.196381297 (2h48m47.977314675s) [upload=2588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.034966654s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X.XX....X...X.................XX....X......XX.XXXX...X..] duration: 953.943188ms FAIL (2025-07-27 00:40:04 UTC) orbiter warning: context deadline exceeded Duration: 953.975457ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 201.559611ms FAIL (2025-07-27 00:39:48 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 201.804982ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.024334636s FAIL (2025-07-27 00:39:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 080 (2h47m2.580185462s) [noise=true , address=77.90.31.21:4896 , piece_id=COXDKGG7XNR3P7TVRMGWYJDNGNEC3ZH7QNERPIYPHTAHFQAVPYGQ, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.799618326 (2h47m2.580551708s) [noise=true , address=64.49.13.139:4478 , piece_id=OERSQCV2O7HWGSN7AKBOBUWJ7CJAPCY6K4ZBO3APZNYRQP4PGZZA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.800046236 (2h47m2.58097962s) [noise=true , address=77.90.31.17:4781 , piece_id=NB2CVWWPMSKW7R3ZE7JXG7STBQS34MTP7QW4DTE327R55T7CGBLQ, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.800300253 (2h47m2.581233638s) [noise=true , address=79.127.205.245:4239 , piece_id=UWGA7PDSLE23C73MVUOWTITRP7BM32VYUFNKYIIECE5RN5DYMXPA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.800347041 (2h47m2.581280418s) [noise=true , address=104.36.82.51:4054 , piece_id=PIUVXUJPZFJWDBZQJL4O43YQLJADAGH4CFGTMDBOTUUT6ST7YL5Q, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.800625944 (2h47m2.581559326s) [noise=true , address=64.49.13.140:4508 , piece_id=VHDWMVQOQ6DN2IL3JWRGITQOP65AH7IC2J6Y4ORRYRURPMFHMDCA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.800767034 (2h47m2.581700419s) [noise=true , address=104.36.82.47:4536 , piece_id=R2XN23G3RTRFTP2POOHZZGMB5MCT37GW2E2GVLBPRYU6K2WPEBXA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.800979546 (2h47m2.581912933s) [noise=true , address=64.49.13.137:4502 , piece_id=TLYY5PFEH5MAEVVPIOIIX7YWR2N76VDVCXAXZX3ECQLOJJ4GRESA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.801245206 (2h47m2.582178591s) [noise=true , address=104.36.82.56:4262 , piece_id=QGAI7HYQ3CCPNGXOKWUEFMZBOGH4E272F6FT4M5UHTTJGGNEOW2Q, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.801328717 (2h47m2.582262099s) [noise=true , address=79.127.205.233:4653 , piece_id=KDVOTRYSU5KTUMKI3QCGORJQNTIUL2TCZIIKVTOOICQKSJMKHVSQ, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.801658253 (2h47m2.582591632s) [noise=true , address=79.127.205.234:4613 , piece_id=NASDYC6IZMHC25KDSAD6CHSI3E5CDC6VU4VLACSG2W32VUHBJ2GA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.801691028 (2h47m2.58262441s) [noise=true , address=104.36.82.53:4142 , piece_id=2RYOD4S5ELK5ZRSGIIIXW6VGWTXS2F3PUGD3C4VWYAFZHQELHPAA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.801965214 (2h47m2.582898591s) [noise=true , address=79.127.205.246:4284 , piece_id=ZNGEUIT2PQW2664QB2PKKBGFQL3VL7PWFW2PS7M3VQFEWH6VHZTA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.802152605 (2h47m2.583085987s) [noise=true , address=77.90.31.15:4715 , piece_id=D3X3UAXWEO7EHFYIKDCXHCBL4YNJEBIB477VPK7V6JFBXORK2AQA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.803214344 (2h47m2.584147722s) [noise=true , address=79.127.205.235:4690 , piece_id=UUC5UEO3ZVHD2Y2PZYQXBN3EDMWKGD236ZPB4GVZWMI4TWGRECQA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.803566988 (2h47m2.584500367s) [noise=true , address=104.36.82.52:4111 , piece_id=XA6FW5I37B6533HP3SKFNWJM5F6VCRPVRRWDAF3IWQJMV6AUFL2Q, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.806582946 (2h47m2.587516337s) [noise=true , address=104.36.82.55:4212 , piece_id=5T4IJFZDKQIK7SWNATMLB67YUPIXB2EBZ4RSDFAVWGTUPL6XMTDA, seg_pos={0 0}, upload=2570]: Uploading piece... Jul 27 00:38:46.876021338 (2h47m2.656954703s) [upload=2570]: Next returned nil segment Jul 27 00:38:46.876037884 (2h47m2.656971259s) [upload=2570]: Waiting for error group managing segments... Jul 27 00:38:47.413224309 (2h47m3.194157697s) [noise=true , address=79.127.205.233:4653 , piece_id=KDVOTRYSU5KTUMKI3QCGORJQNTIUL2TCZIIKVTOOICQKSJMKHVSQ, seg_pos={0 0}, upload=2570]: Done uploading piece. err: <nil> Jul 27 00:38:47.447700688 (2h47m3.228634054s) [noise=true , address=64.49.13.137:4502 , piece_id=TLYY5PFEH5MAEVVPIOIIX7YWR2N76VDVCXAXZX3ECQLOJJ4GRESA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:47.469368277 (2h47m3.250301645s) [noise=true , address=79.127.205.234:4613 , piece_id=NASDYC6IZMHC25KDSAD6CHSI3E5CDC6VU4VLACSG2W32VUHBJ2GA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: <nil> Jul 27 00:38:47.492983432 (2h47m3.2739168s) [noise=true , address=79.127.205.245:4239 , piece_id=UWGA7PDSLE23C73MVUOWTITRP7BM32VYUFNKYIIECE5RN5DYMXPA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: <nil> Jul 27 00:38:47.501693952 (2h47m3.28262732s) [noise=true , address=79.127.205.246:4284 , piece_id=ZNGEUIT2PQW2664QB2PKKBGFQL3VL7PWFW2PS7M3VQFEWH6VHZTA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: <nil> Jul 27 00:38:47.605209040 (2h47m3.38614242s) [noise=true , address=79.127.205.235:4690 , piece_id=UUC5UEO3ZVHD2Y2PZYQXBN3EDMWKGD236ZPB4GVZWMI4TWGRECQA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: <nil> Jul 27 00:38:48.563085714 (2h47m4.344019086s) [noise=true , address=77.90.31.17:4781 , piece_id=NB2CVWWPMSKW7R3ZE7JXG7STBQS34MTP7QW4DTE327R55T7CGBLQ, seg_pos={0 0}, upload=2570]: Done uploading piece. err: <nil> Jul 27 00:38:49.213782413 (2h47m4.994715791s) [noise=true , address=64.49.13.139:4478 , piece_id=OERSQCV2O7HWGSN7AKBOBUWJ7CJAPCY6K4ZBO3APZNYRQP4PGZZA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:49.279226244 (2h47m5.06015961s) [noise=true , address=64.49.13.140:4508 , piece_id=VHDWMVQOQ6DN2IL3JWRGITQOP65AH7IC2J6Y4ORRYRURPMFHMDCA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: <nil> Jul 27 00:39:10.577647223 (2h47m26.358580599s) [noise=true , address=79.127.201.215:4344 , piece_id=Z2JXZ5VWYSQ2KTFXXOEIYJLTVFGKH7MQO5KHLZXRA4K32PA4ZKYA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: <nil> Jul 27 00:39:46.731446201 (2h48m2.512379567s) [noise=true , address=104.36.82.51:4054 , piece_id=PIUVXUJPZFJWDBZQJL4O43YQLJADAGH4CFGTMDBOTUUT6ST7YL5Q, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v, address:104.36.82.51:4054): protocol: rpc: dial tcp 104.36.82.51:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:46.731508634 (2h48m2.512441999s) [noise=true , address=104.36.82.52:4111 , piece_id=XA6FW5I37B6533HP3SKFNWJM5F6VCRPVRRWDAF3IWQJMV6AUFL2Q, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: rpc: dial tcp 104.36.82.52:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:46.731556674 (2h48m2.512490038s) [noise=true , address=104.36.82.55:4212 , piece_id=5T4IJFZDKQIK7SWNATMLB67YUPIXB2EBZ4RSDFAVWGTUPL6XMTDA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: rpc: dial tcp 104.36.82.55:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:46.731606839 (2h48m2.512540205s) [noise=true , address=77.90.31.15:4715 , piece_id=D3X3UAXWEO7EHFYIKDCXHCBL4YNJEBIB477VPK7V6JFBXORK2AQA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:46.731651651 (2h48m2.512585017s) [noise=true , address=104.36.82.47:4536 , piece_id=R2XN23G3RTRFTP2POOHZZGMB5MCT37GW2E2GVLBPRYU6K2WPEBXA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: rpc: dial tcp 104.36.82.47:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:46.731756180 (2h48m2.512689553s) [noise=true , address=104.36.82.56:4262 , piece_id=QGAI7HYQ3CCPNGXOKWUEFMZBOGH4E272F6FT4M5UHTTJGGNEOW2Q, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:1DY2kpdCo8EDSs1LHL6qBFo6qjnsb7FL1QDekMV5qaWB5jxeMz, address:104.36.82.56:4262): protocol: rpc: dial tcp 104.36.82.56:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:46.731823939 (2h48m2.512757303s) [noise=true , address=104.36.82.53:4142 , piece_id=2RYOD4S5ELK5ZRSGIIIXW6VGWTXS2F3PUGD3C4VWYAFZHQELHPAA, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: rpc: dial tcp 104.36.82.53:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:46.731879369 (2h48m2.512812737s) [noise=true , address=77.90.31.21:4896 , piece_id=COXDKGG7XNR3P7TVRMGWYJDNGNEC3ZH7QNERPIYPHTAHFQAVPYGQ, seg_pos={0 0}, upload=2570]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: read tcp 159.223.143.39:59154->77.90.31.21:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:39:46.731903131 (2h48m2.512836497s) [seg_pos={0 0}, upload=2570]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:39:46.753790606 (2h48m2.534723982s) [upload=2570]: Done uploading segments... Jul 27 00:39:46.753817265 (2h48m2.534750641s) [upload=2570]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.02437517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX.] duration: 699.011101ms FAIL (2025-07-27 00:39:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 49445 (2h46m40.141482829s) [noise=true , address=104.36.82.2:4729 , piece_id=LA7Z25NVUVE4MWYV6HLIVOOUO4HXJG373E25LUP42K7OYPMMLMFQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.360804130 (2h46m40.141737508s) [noise=true , address=45.91.116.13:4153 , piece_id=WXPCCBOPAZ4ETRIE3OBAUC75COMWBZDY5UEK3E6DMU4GNXAQBGAA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.361073036 (2h46m40.142006421s) [noise=true , address=104.36.82.15:4258 , piece_id=CHEGB4TEC5PQWHQIZGVP2S3T3XO3BQY4EEZJSFM3T6GRPRNHRRQQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.361450252 (2h46m40.142383632s) [noise=true , address=104.36.82.6:4874 , piece_id=U6DO42NOHSYS3ECLMUIFRB4Z4FR5LQWNXT2I4ZQX4JA57GQMIEXA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.361549731 (2h46m40.142483118s) [noise=true , address=104.36.82.7:4912 , piece_id=2D6VVHMLYYBH63ARVP44GVFMGEB7UWUG3WIWDI4G36D3HK7ST6MA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362015372 (2h46m40.14294876s) [noise=true , address=104.36.82.14:4222 , piece_id=YA6IS2XVYZW6JE6RO4AN2NU56AH5XIY3ONUVTIH3IQTG2TPHG6YQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362090019 (2h46m40.143023398s) [noise=true , address=45.91.116.8:4045 , piece_id=Y7CJKBWOBGDNZE2BYZ7PDTLUSOZCHC6E6QH3V54DOYH3TBPZ2XEA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362455614 (2h46m40.143389001s) [noise=true , address=104.36.82.13:4187 , piece_id=GY2KFCGUIHNKTNMIH7KHJTKO7OHAIJJXNSOUJQXIINKTVWVPXNZQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362822950 (2h46m40.143756328s) [noise=true , address=77.90.31.21:4878 , piece_id=MR4HYP7NFKF2EN5C2ZVO6THFB2GNHYKZNQHSIF7GCT56ZZN7FYEQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362874615 (2h46m40.143808005s) [noise=true , address=104.36.82.4:4801 , piece_id=T35S3323YUF6ZNVES4UZV2E2RB7T477QPPZMF2K7AF2XWX3Z744Q, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.363397122 (2h46m40.144330507s) [noise=true , address=77.90.31.13:4628 , piece_id=MAHJYWJ6ZZYU5BNZRX2BAQPAHI3XY6BSPL2L4U34UYOPMCC5TCWA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.363906161 (2h46m40.144839544s) [noise=true , address=104.36.82.6:4876 , piece_id=XUQ4UKGYVEZNGDF5VL4D2XZGCV7PRMK5LY5HLEIQXMZ3IS7YGPNA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.364333554 (2h46m40.145266937s) [noise=true , address=104.36.82.12:4149 , piece_id=FZAB4KJUADVLKLOZA4BYT2DRYZYIWRFV3A5E2HW4RUBNBO5IYHIQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.364797876 (2h46m40.145731262s) [noise=true , address=104.36.82.10:4079 , piece_id=ERRNE5US45RORAV6SGN7YZZ4VBYMTR6MQJMDD7B3676VVDMUJBLA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.461464273 (2h46m40.242397638s) [upload=2565]: Next returned nil segment Jul 27 00:38:24.461482392 (2h46m40.242415768s) [upload=2565]: Waiting for error group managing segments... Jul 27 00:38:24.466252123 (2h46m40.247185502s) [noise=true , address=45.91.116.13:4153 , piece_id=WXPCCBOPAZ4ETRIE3OBAUC75COMWBZDY5UEK3E6DMU4GNXAQBGAA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: <nil> Jul 27 00:38:24.466567455 (2h46m40.247500824s) [noise=true , address=77.90.31.16:4733 , piece_id=2JOHJMKQ5ROWU3CBLDUOTPWJKMYY2WGAFSHOOOSPD4O5PJTE5UAA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: <nil> Jul 27 00:38:24.467008241 (2h46m40.247941619s) [noise=true , address=45.91.116.8:4045 , piece_id=Y7CJKBWOBGDNZE2BYZ7PDTLUSOZCHC6E6QH3V54DOYH3TBPZ2XEA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: <nil> Jul 27 00:38:25.861348937 (2h46m41.642282313s) [noise=true , address=77.90.31.13:4628 , piece_id=MAHJYWJ6ZZYU5BNZRX2BAQPAHI3XY6BSPL2L4U34UYOPMCC5TCWA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: <nil> Jul 27 00:38:54.300639043 (2h47m10.081572428s) [noise=true , address=104.36.82.6:4874 , piece_id=U6DO42NOHSYS3ECLMUIFRB4Z4FR5LQWNXT2I4ZQX4JA57GQMIEXA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:45306->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.300744240 (2h47m10.081677615s) [noise=true , address=104.36.82.15:4258 , piece_id=CHEGB4TEC5PQWHQIZGVP2S3T3XO3BQY4EEZJSFM3T6GRPRNHRRQQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:42086->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.300795695 (2h47m10.081729059s) [noise=true , address=77.90.31.21:4878 , piece_id=MR4HYP7NFKF2EN5C2ZVO6THFB2GNHYKZNQHSIF7GCT56ZZN7FYEQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:41010->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.300853698 (2h47m10.081787063s) [noise=true , address=104.36.82.10:4079 , piece_id=ERRNE5US45RORAV6SGN7YZZ4VBYMTR6MQJMDD7B3676VVDMUJBLA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:50230->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301156174 (2h47m10.082089549s) [noise=true , address=104.36.82.13:4187 , piece_id=GY2KFCGUIHNKTNMIH7KHJTKO7OHAIJJXNSOUJQXIINKTVWVPXNZQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:36046->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301211772 (2h47m10.082145138s) [noise=true , address=104.36.82.7:4912 , piece_id=2D6VVHMLYYBH63ARVP44GVFMGEB7UWUG3WIWDI4G36D3HK7ST6MA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:42894->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301410947 (2h47m10.082344322s) [noise=true , address=104.36.82.12:4149 , piece_id=FZAB4KJUADVLKLOZA4BYT2DRYZYIWRFV3A5E2HW4RUBNBO5IYHIQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:33300->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301452350 (2h47m10.082385718s) [noise=true , address=104.36.82.14:4222 , piece_id=YA6IS2XVYZW6JE6RO4AN2NU56AH5XIY3ONUVTIH3IQTG2TPHG6YQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:38596->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301480956 (2h47m10.082414332s) [noise=true , address=77.90.31.15:4697 , piece_id=I2EVAKTM7CRLVJD7OLWCYPNTFY7CHZNSISAJQCQINQDXM4GP4DKQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:46632->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301517702 (2h47m10.082451068s) [noise=true , address=104.36.82.6:4876 , piece_id=XUQ4UKGYVEZNGDF5VL4D2XZGCV7PRMK5LY5HLEIQXMZ3IS7YGPNA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:46848->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301544308 (2h47m10.082477683s) [noise=true , address=104.36.82.4:4801 , piece_id=T35S3323YUF6ZNVES4UZV2E2RB7T477QPPZMF2K7AF2XWX3Z744Q, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:44826->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301568340 (2h47m10.082501707s) [noise=true , address=104.36.82.2:4729 , piece_id=LA7Z25NVUVE4MWYV6HLIVOOUO4HXJG373E25LUP42K7OYPMMLMFQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:54626->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301593952 (2h47m10.08252733s) [seg_pos={0 0}, upload=2565]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Jul 27 00:38:54.323451208 (2h47m10.104384605s) [upload=2565]: Done uploading segments... Jul 27 00:38:54.323480163 (2h47m10.104413529s) [upload=2565]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 699.049085ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X...XX...............................X...............X...] duration: 48.254908912s FAIL (2025-07-27 00:39:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.255000335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX...] duration: 1.892044634s FAIL (2025-07-27 00:39:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.892109717s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 425.453447ms FAIL (2025-07-27 00:39:23 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 425.484466ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX..] duration: 1.85117145s FAIL (2025-07-27 00:39:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.851204153s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X.XX....X...X.................XX....X......XX.XXXX...X.] duration: 7.028423892s FAIL (2025-07-27 00:39:00 UTC) orbiter warning: context deadline exceeded Duration: 7.028445921s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XXX] duration: 30.026456696s FAIL (2025-07-27 00:38:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 49445 (2h46m40.141482829s) [noise=true , address=104.36.82.2:4729 , piece_id=LA7Z25NVUVE4MWYV6HLIVOOUO4HXJG373E25LUP42K7OYPMMLMFQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.360804130 (2h46m40.141737508s) [noise=true , address=45.91.116.13:4153 , piece_id=WXPCCBOPAZ4ETRIE3OBAUC75COMWBZDY5UEK3E6DMU4GNXAQBGAA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.361073036 (2h46m40.142006421s) [noise=true , address=104.36.82.15:4258 , piece_id=CHEGB4TEC5PQWHQIZGVP2S3T3XO3BQY4EEZJSFM3T6GRPRNHRRQQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.361450252 (2h46m40.142383632s) [noise=true , address=104.36.82.6:4874 , piece_id=U6DO42NOHSYS3ECLMUIFRB4Z4FR5LQWNXT2I4ZQX4JA57GQMIEXA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.361549731 (2h46m40.142483118s) [noise=true , address=104.36.82.7:4912 , piece_id=2D6VVHMLYYBH63ARVP44GVFMGEB7UWUG3WIWDI4G36D3HK7ST6MA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362015372 (2h46m40.14294876s) [noise=true , address=104.36.82.14:4222 , piece_id=YA6IS2XVYZW6JE6RO4AN2NU56AH5XIY3ONUVTIH3IQTG2TPHG6YQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362090019 (2h46m40.143023398s) [noise=true , address=45.91.116.8:4045 , piece_id=Y7CJKBWOBGDNZE2BYZ7PDTLUSOZCHC6E6QH3V54DOYH3TBPZ2XEA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362455614 (2h46m40.143389001s) [noise=true , address=104.36.82.13:4187 , piece_id=GY2KFCGUIHNKTNMIH7KHJTKO7OHAIJJXNSOUJQXIINKTVWVPXNZQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362822950 (2h46m40.143756328s) [noise=true , address=77.90.31.21:4878 , piece_id=MR4HYP7NFKF2EN5C2ZVO6THFB2GNHYKZNQHSIF7GCT56ZZN7FYEQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.362874615 (2h46m40.143808005s) [noise=true , address=104.36.82.4:4801 , piece_id=T35S3323YUF6ZNVES4UZV2E2RB7T477QPPZMF2K7AF2XWX3Z744Q, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.363397122 (2h46m40.144330507s) [noise=true , address=77.90.31.13:4628 , piece_id=MAHJYWJ6ZZYU5BNZRX2BAQPAHI3XY6BSPL2L4U34UYOPMCC5TCWA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.363906161 (2h46m40.144839544s) [noise=true , address=104.36.82.6:4876 , piece_id=XUQ4UKGYVEZNGDF5VL4D2XZGCV7PRMK5LY5HLEIQXMZ3IS7YGPNA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.364333554 (2h46m40.145266937s) [noise=true , address=104.36.82.12:4149 , piece_id=FZAB4KJUADVLKLOZA4BYT2DRYZYIWRFV3A5E2HW4RUBNBO5IYHIQ, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.364797876 (2h46m40.145731262s) [noise=true , address=104.36.82.10:4079 , piece_id=ERRNE5US45RORAV6SGN7YZZ4VBYMTR6MQJMDD7B3676VVDMUJBLA, seg_pos={0 0}, upload=2565]: Uploading piece... Jul 27 00:38:24.461464273 (2h46m40.242397638s) [upload=2565]: Next returned nil segment Jul 27 00:38:24.461482392 (2h46m40.242415768s) [upload=2565]: Waiting for error group managing segments... Jul 27 00:38:24.466252123 (2h46m40.247185502s) [noise=true , address=45.91.116.13:4153 , piece_id=WXPCCBOPAZ4ETRIE3OBAUC75COMWBZDY5UEK3E6DMU4GNXAQBGAA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: <nil> Jul 27 00:38:24.466567455 (2h46m40.247500824s) [noise=true , address=77.90.31.16:4733 , piece_id=2JOHJMKQ5ROWU3CBLDUOTPWJKMYY2WGAFSHOOOSPD4O5PJTE5UAA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: <nil> Jul 27 00:38:24.467008241 (2h46m40.247941619s) [noise=true , address=45.91.116.8:4045 , piece_id=Y7CJKBWOBGDNZE2BYZ7PDTLUSOZCHC6E6QH3V54DOYH3TBPZ2XEA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: <nil> Jul 27 00:38:25.861348937 (2h46m41.642282313s) [noise=true , address=77.90.31.13:4628 , piece_id=MAHJYWJ6ZZYU5BNZRX2BAQPAHI3XY6BSPL2L4U34UYOPMCC5TCWA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: <nil> Jul 27 00:38:54.300639043 (2h47m10.081572428s) [noise=true , address=104.36.82.6:4874 , piece_id=U6DO42NOHSYS3ECLMUIFRB4Z4FR5LQWNXT2I4ZQX4JA57GQMIEXA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:45306->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.300744240 (2h47m10.081677615s) [noise=true , address=104.36.82.15:4258 , piece_id=CHEGB4TEC5PQWHQIZGVP2S3T3XO3BQY4EEZJSFM3T6GRPRNHRRQQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:42086->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.300795695 (2h47m10.081729059s) [noise=true , address=77.90.31.21:4878 , piece_id=MR4HYP7NFKF2EN5C2ZVO6THFB2GNHYKZNQHSIF7GCT56ZZN7FYEQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:41010->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.300853698 (2h47m10.081787063s) [noise=true , address=104.36.82.10:4079 , piece_id=ERRNE5US45RORAV6SGN7YZZ4VBYMTR6MQJMDD7B3676VVDMUJBLA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:50230->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301156174 (2h47m10.082089549s) [noise=true , address=104.36.82.13:4187 , piece_id=GY2KFCGUIHNKTNMIH7KHJTKO7OHAIJJXNSOUJQXIINKTVWVPXNZQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:36046->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301211772 (2h47m10.082145138s) [noise=true , address=104.36.82.7:4912 , piece_id=2D6VVHMLYYBH63ARVP44GVFMGEB7UWUG3WIWDI4G36D3HK7ST6MA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:42894->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301410947 (2h47m10.082344322s) [noise=true , address=104.36.82.12:4149 , piece_id=FZAB4KJUADVLKLOZA4BYT2DRYZYIWRFV3A5E2HW4RUBNBO5IYHIQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:33300->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301452350 (2h47m10.082385718s) [noise=true , address=104.36.82.14:4222 , piece_id=YA6IS2XVYZW6JE6RO4AN2NU56AH5XIY3ONUVTIH3IQTG2TPHG6YQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:38596->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301480956 (2h47m10.082414332s) [noise=true , address=77.90.31.15:4697 , piece_id=I2EVAKTM7CRLVJD7OLWCYPNTFY7CHZNSISAJQCQINQDXM4GP4DKQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:46632->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301517702 (2h47m10.082451068s) [noise=true , address=104.36.82.6:4876 , piece_id=XUQ4UKGYVEZNGDF5VL4D2XZGCV7PRMK5LY5HLEIQXMZ3IS7YGPNA, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:46848->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301544308 (2h47m10.082477683s) [noise=true , address=104.36.82.4:4801 , piece_id=T35S3323YUF6ZNVES4UZV2E2RB7T477QPPZMF2K7AF2XWX3Z744Q, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:44826->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301568340 (2h47m10.082501707s) [noise=true , address=104.36.82.2:4729 , piece_id=LA7Z25NVUVE4MWYV6HLIVOOUO4HXJG373E25LUP42K7OYPMMLMFQ, seg_pos={0 0}, upload=2565]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:54626->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:54.301593952 (2h47m10.08252733s) [seg_pos={0 0}, upload=2565]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Jul 27 00:38:54.323451208 (2h47m10.104384605s) [upload=2565]: Done uploading segments... Jul 27 00:38:54.323480163 (2h47m10.104413529s) [upload=2565]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.026475129s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X.XX....X...X.................XX....X......XX.XXXX...X] duration: 30.001498896s FAIL (2025-07-27 00:38:51 UTC) orbiter warning: context deadline exceeded Duration: 30.001525947s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030514429s FAIL (2025-07-27 00:38:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== h45m59.648500711s) [seg_pos={0 0}, upload=2552]: Begin upload segment... Jul 27 00:37:43.867621208 (2h45m59.648554577s) [seg_pos={0 0}, upload=2552]: Done begin upload segment. Jul 27 00:37:43.867652184 (2h45m59.648585561s) [noise=true , address=77.90.31.12:4608 , piece_id=BHVT6HOZIN3QX7QRWD66444A63XLEWW4YMVCBPSWE46DYALRVHUA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.867695182 (2h45m59.648628558s) [noise=true , address=104.36.82.47:4544 , piece_id=REDHULPDXJH2M7HLN5G7YYR2O3LMPJFZY3KMTXY4YL52CUE2DSCA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.868192585 (2h45m59.649125971s) [noise=true , address=104.36.82.52:4111 , piece_id=PTDXPUMTOXT3UUBNVRH6GWIDSFZW7T5UBN5SOOJESU3SJ2BNH3OA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.868400914 (2h45m59.64933429s) [noise=true , address=77.90.31.18:4821 , piece_id=O6GIO4BF5FXPNDPZ3BPAD6Y2X4V4QGO5OVKTL7HY5LTMEWPLVBJQ, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.868489626 (2h45m59.649423s) [noise=true , address=77.90.31.11:4584 , piece_id=3EM37AXPMEBDZ7WRXFZZ5C7ZSEQQL6WS3JOTAOBXWOOTMNW7OHDA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.868797285 (2h45m59.649730659s) [noise=true , address=79.127.205.235:4688 , piece_id=YKZQGI44R2U6TRTWYC465EGZO2XLNCI2OOIIGKPP3332UIFBFYDA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.869083966 (2h45m59.650017347s) [noise=true , address=104.36.82.53:4126 , piece_id=7WASNMDF4ORANY4R7UCDYQVE76YUPU257TDQ52CCWG22TWAXO6PQ, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.870199350 (2h45m59.65113273s) [noise=true , address=45.91.116.10:4093 , piece_id=4SCLRSBVXI2KJOTDYSWZZG3FQGV66AJUQC275SKS7ISBPBKF7QSA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.870779049 (2h45m59.651712429s) [noise=true , address=45.91.116.14:4206 , piece_id=OJJYJFDMAMZBWAVKZIXZRV3RQHJKXI7ONXTUKLN2FDWFWDDSGN5A, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.871695610 (2h45m59.652628976s) [noise=true , address=79.127.201.215:4346 , piece_id=IKRHENM6FTANOBABG2X24PKO7P2T5ZBCR4NX3UWMTX6LGA72NHOA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.873005891 (2h45m59.653939288s) [noise=true , address=104.36.82.46:4491 , piece_id=A7QF2XGS2OUYN2RZH5X4723JBD4HCHHWY66ZCO2NUBFR73N4IAQQ, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.873849494 (2h45m59.654782884s) [noise=true , address=104.36.82.3:4785 , piece_id=APSRBRDJKRZOHVKIUWNP5UXO3GVEQCNDBIIRKTNOD2SRW5S3GRLQ, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.874464543 (2h45m59.655397927s) [noise=true , address=156.146.43.225:4461 , piece_id=OHXEZDWEGQ7SDGQXQPAPAE44JVCT3ISIBC6ITHF6QYKBPL5XXLMA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.874948712 (2h45m59.655882089s) [noise=true , address=45.91.116.4:4964 , piece_id=DDVKTNLO4GJEUQPGWN7PNZJPVLOCZBMMRFUDTYHUROAX65H3OXKQ, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.875413232 (2h45m59.656346615s) [noise=true , address=77.90.31.17:4793 , piece_id=5VHGBMBNDNTQMDNKG3KL2NIAFEQD3EE4IUHFTPUF6AIPOOZ24YSA, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.875879054 (2h45m59.656812431s) [noise=true , address=79.127.219.38:4009 , piece_id=G245TRABO6HURYGQFBS3XSOBEOM63CJTJEUUCJNZ32SNH6ESEDKQ, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.876885894 (2h45m59.657819276s) [noise=true , address=104.36.82.6:4893 , piece_id=OUYGGOWIU4N3D6LQP6TPMTICJS4QOZYVMWJK4X53UKZ7TNHPQASQ, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.877296769 (2h45m59.65823013s) [noise=true , address=79.127.201.214:4321 , piece_id=JVHXSPTWG2AI2WULRDLEORDI53LU3MQQ2TYC5NWN42JIA4JUKUJQ, seg_pos={0 0}, upload=2552]: Uploading piece... Jul 27 00:37:43.945394306 (2h45m59.726327692s) [upload=2552]: Next returned nil segment Jul 27 00:37:43.945428568 (2h45m59.726361947s) [upload=2552]: Waiting for error group managing segments... Jul 27 00:37:43.960911779 (2h45m59.741845148s) [noise=true , address=79.127.219.38:4009 , piece_id=G245TRABO6HURYGQFBS3XSOBEOM63CJTJEUUCJNZ32SNH6ESEDKQ, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:37:44.008478004 (2h45m59.789411387s) [noise=true , address=79.127.201.214:4321 , piece_id=JVHXSPTWG2AI2WULRDLEORDI53LU3MQQ2TYC5NWN42JIA4JUKUJQ, seg_pos={0 0}, upload=2552]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 27 00:37:44.019245572 (2h45m59.800178954s) [noise=true , address=156.146.43.225:4461 , piece_id=OHXEZDWEGQ7SDGQXQPAPAE44JVCT3ISIBC6ITHF6QYKBPL5XXLMA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: ecclient: upload failed (node:1D4k7wnNtivVkHko5CSBuZH6zqMuFyfFfshkZwEPqnmpLads49, address:156.146.43.225:4461): protocol: storage node overloaded, request limit: 2000 Jul 27 00:37:44.031305352 (2h45m59.812238715s) [noise=true , address=77.90.31.17:4793 , piece_id=5VHGBMBNDNTQMDNKG3KL2NIAFEQD3EE4IUHFTPUF6AIPOOZ24YSA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:37:44.342015188 (2h46m0.122948556s) [noise=true , address=77.90.31.11:4584 , piece_id=3EM37AXPMEBDZ7WRXFZZ5C7ZSEQQL6WS3JOTAOBXWOOTMNW7OHDA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:37:44.461236727 (2h46m0.242170096s) [noise=true , address=79.127.205.235:4688 , piece_id=YKZQGI44R2U6TRTWYC465EGZO2XLNCI2OOIIGKPP3332UIFBFYDA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:37:45.256803805 (2h46m1.037737171s) [noise=true , address=77.90.31.12:4608 , piece_id=BHVT6HOZIN3QX7QRWD66444A63XLEWW4YMVCBPSWE46DYALRVHUA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:37:45.469761098 (2h46m1.250694465s) [noise=true , address=45.91.116.4:4964 , piece_id=DDVKTNLO4GJEUQPGWN7PNZJPVLOCZBMMRFUDTYHUROAX65H3OXKQ, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:37:49.002035920 (2h46m4.782969302s) [noise=true , address=45.91.116.14:4206 , piece_id=OJJYJFDMAMZBWAVKZIXZRV3RQHJKXI7ONXTUKLN2FDWFWDDSGN5A, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:37:59.260510823 (2h46m15.041444192s) [noise=true , address=79.127.201.215:4346 , piece_id=IKRHENM6FTANOBABG2X24PKO7P2T5ZBCR4NX3UWMTX6LGA72NHOA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:37:59.930822323 (2h46m15.711755708s) [noise=true , address=45.91.116.10:4093 , piece_id=4SCLRSBVXI2KJOTDYSWZZG3FQGV66AJUQC275SKS7ISBPBKF7QSA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:38:00.917674428 (2h46m16.698607795s) [noise=true , address=77.90.31.18:4821 , piece_id=O6GIO4BF5FXPNDPZ3BPAD6Y2X4V4QGO5OVKTL7HY5LTMEWPLVBJQ, seg_pos={0 0}, upload=2552]: Done uploading piece. err: <nil> Jul 27 00:38:43.786018481 (2h46m59.566951865s) [noise=true , address=104.36.82.53:4126 , piece_id=7WASNMDF4ORANY4R7UCDYQVE76YUPU257TDQ52CCWG22TWAXO6PQ, seg_pos={0 0}, upload=2552]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: rpc: dial tcp 104.36.82.53:4126: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:43.786177159 (2h46m59.567110543s) [noise=true , address=104.36.82.46:4491 , piece_id=A7QF2XGS2OUYN2RZH5X4723JBD4HCHHWY66ZCO2NUBFR73N4IAQQ, seg_pos={0 0}, upload=2552]: Done uploading piece. err: ecclient: upload failed (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX, address:104.36.82.46:4491): protocol: rpc: dial tcp 104.36.82.46:4491: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:43.786312621 (2h46m59.567246001s) [noise=true , address=104.36.82.6:4893 , piece_id=OUYGGOWIU4N3D6LQP6TPMTICJS4QOZYVMWJK4X53UKZ7TNHPQASQ, seg_pos={0 0}, upload=2552]: Done uploading piece. err: ecclient: upload failed (node:1259j7aHsLDMPtRwgXpd6u8axPEuaXzhRzqYYSgbpDnDqbhfwzi, address:104.36.82.6:4893): protocol: rpc: dial tcp 104.36.82.6:4893: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:43.786368828 (2h46m59.567302211s) [noise=true , address=104.36.82.47:4544 , piece_id=REDHULPDXJH2M7HLN5G7YYR2O3LMPJFZY3KMTXY4YL52CUE2DSCA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:104.36.82.47:4544): protocol: write tcp 159.223.143.39:59838->104.36.82.47:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:43.786391254 (2h46m59.567324743s) [noise=true , address=104.36.82.52:4111 , piece_id=PTDXPUMTOXT3UUBNVRH6GWIDSFZW7T5UBN5SOOJESU3SJ2BNH3OA, seg_pos={0 0}, upload=2552]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: rpc: dial tcp 104.36.82.52:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:43.786448817 (2h46m59.567382206s) [noise=true , address=104.36.82.3:4785 , piece_id=APSRBRDJKRZOHVKIUWNP5UXO3GVEQCNDBIIRKTNOD2SRW5S3GRLQ, seg_pos={0 0}, upload=2552]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:104.36.82.3:4785): protocol: rpc: dial tcp 104.36.82.3:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:43.786496525 (2h46m59.567429912s) [seg_pos={0 0}, upload=2552]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:38:43.812686613 (2h46m59.593619984s) [upload=2552]: Done uploading segments... Jul 27 00:38:43.812765277 (2h46m59.593698653s) [upload=2552]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.030598742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX.] duration: 1.820471441s FAIL (2025-07-27 00:38:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.820505559s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXXX] duration: 30.02498543s FAIL (2025-07-27 00:38:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025028043s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..XX] duration: 30.037650722s FAIL (2025-07-27 00:38:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 77.90.31.15:4698 , piece_id=R7BVEFUKRNJARZVBW3WGQVE25STNEFPWUXGINXDEIVWTHLNTJX5Q, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 159.223.143.39:49862->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217865608 (2h45m34.998798994s) [seg_pos={0 0}, upload=2538]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:37:19.241912345 (2h45m35.022845714s) [upload=2538]: Done uploading segments... Jul 27 00:37:19.241938238 (2h45m35.022871613s) [upload=2538]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:37:54.224553875 (2h46m10.005487253s) [upload=2554]: Starting upload Jul 27 00:37:54.224624908 (2h46m10.005558284s) [upload=2554]: Done starting upload Jul 27 00:37:54.224732962 (2h46m10.005666341s) [upload=2554]: Uploading segments... Jul 27 00:37:54.224939269 (2h46m10.005872655s) [upload=2554]: Got next segment. Inline: false Jul 27 00:37:54.224994912 (2h46m10.005928293s) [upload=2554]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:37:54.225021264 (2h46m10.005954643s) [upload=2554]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:37:54.264793429 (2h46m10.045726796s) [seg_pos={0 0}, upload=2554]: Begin upload segment... Jul 27 00:37:54.264853130 (2h46m10.045786496s) [seg_pos={0 0}, upload=2554]: Done begin upload segment. Jul 27 00:37:54.264878810 (2h46m10.045812188s) [noise=true , address=45.91.116.12:4227 , piece_id=RNL7DTYKKATXOBE76D523ENDIHCG3QRUXKHNKWETAOZAHCK7GP7Q, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.265134814 (2h46m10.046068182s) [noise=true , address=45.91.116.11:4118 , piece_id=IQ5CLCX7FYP7NOJ455WEPJUAG3INODD7LAYI2B66EZ5PWOQMZW2A, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.265633177 (2h46m10.046566557s) [noise=true , address=77.90.31.16:4735 , piece_id=YADS3OVIRUKMYWHURDC53FA2Y4KGZFPXMKGDQFYBEHIWPBDJQWQA, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.266675651 (2h46m10.047609021s) [noise=true , address=77.90.31.21:4879 , piece_id=6JOP3SKMWUGQJSFJESPBN35SHIGO4A3XU5SWXUBYHHFO3QFUXE7Q, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.267435985 (2h46m10.048369354s) [noise=true , address=104.36.82.10:4077 , piece_id=UQWHPVJHCFJWBXTWGFC54RB4N2HIWGTYID4FOJA5Q6MM3PESEM5A, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.267679026 (2h46m10.048612388s) [noise=true , address=45.91.116.6:4975 , piece_id=RSKNZO5PGDGPJHSJG5O37M2DUTOKPSYNBMXMMPYVO5BXFBVT5XAQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.268208764 (2h46m10.049142133s) [noise=true , address=45.91.116.10:4082 , piece_id=GOA4OFIHC7V4VSOSXQVJFNBNF5LDFENYV3VV4DTOAZUTWZYFDIEQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.268560300 (2h46m10.049493669s) [noise=true , address=77.90.31.17:4772 , piece_id=I4DSQONAAV4ILKJTSESSHAVRZ3GFG7STRCVLNHLCYSCP3X5S2UPQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.268846706 (2h46m10.049780074s) [noise=true , address=45.91.116.14:4190 , piece_id=DLWQJSE64S5KGO6LR4NCY6K3FS5FPSFGV5AMLJQ6N33Z6VDQM5UQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.268976931 (2h46m10.049910295s) [noise=true , address=104.36.82.4:4803 , piece_id=FFUM4NUBYWJYYUP5RWZJ5BSFNU5RLQIIGGO3MXEYG2TMPDZQYPSA, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.269413216 (2h46m10.050346579s) [noise=true , address=104.36.82.7:4909 , piece_id=VBLVOC6DNKP6I5URHWF5IUOXWA7CEPYPFETOF47JSHPCCET7VAEQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.269523128 (2h46m10.050456496s) [noise=true , address=77.90.31.14:4663 , piece_id=2CWHV3ZN5XCRVQ3ZCWNYL7D5JRMWLCMEN5GNLGHYDZXVVYBC4QIQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.269873949 (2h46m10.050807317s) [noise=true , address=77.90.31.15:4699 , piece_id=N6NQF3AOROQJFFRXUCYNRSHUI7XB5XNRHGSS4SWFPIVSUUNJAVZQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.270118116 (2h46m10.051051483s) [noise=true , address=45.91.116.5:4298 , piece_id=OLJLZN3VOCEBW5GIPNPHJ6GHNW25O3HW6ALAF7FLEJCHLPAWWPTQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.270564433 (2h46m10.051497802s) [noise=true , address=45.91.116.13:4155 , piece_id=DEZRUTXKADKW3OYSW6N5WZ4RIL3X5S3VNDMAN5AAQZN4J4D23Q5A, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.270907864 (2h46m10.051841232s) [noise=true , address=45.91.116.9:4263 , piece_id=SHN7ZV6VQJ2FB26R3BIRO7O3YHWME7W4WZY4I22C7J4ALNRTZADQ, seg_pos={0 0}, upload=2554]: Uploading piece... Jul 27 00:37:54.375518751 (2h46m10.156452131s) [upload=2554]: Next returned nil segment Jul 27 00:37:54.375543795 (2h46m10.156477185s) [upload=2554]: Waiting for error group managing segments... Jul 27 00:37:54.379496221 (2h46m10.160429597s) [noise=true , address=45.91.116.13:4155 , piece_id=DEZRUTXKADKW3OYSW6N5WZ4RIL3X5S3VNDMAN5AAQZN4J4D23Q5A, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.379887938 (2h46m10.160821316s) [noise=true , address=45.91.116.5:4298 , piece_id=OLJLZN3VOCEBW5GIPNPHJ6GHNW25O3HW6ALAF7FLEJCHLPAWWPTQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.380175801 (2h46m10.161109163s) [noise=true , address=45.91.116.12:4227 , piece_id=RNL7DTYKKATXOBE76D523ENDIHCG3QRUXKHNKWETAOZAHCK7GP7Q, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.380563419 (2h46m10.161496796s) [noise=true , address=45.91.116.6:4975 , piece_id=RSKNZO5PGDGPJHSJG5O37M2DUTOKPSYNBMXMMPYVO5BXFBVT5XAQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.380773154 (2h46m10.161706534s) [noise=true , address=45.91.116.11:4118 , piece_id=IQ5CLCX7FYP7NOJ455WEPJUAG3INODD7LAYI2B66EZ5PWOQMZW2A, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.381166882 (2h46m10.162100259s) [noise=true , address=77.90.31.14:4663 , piece_id=2CWHV3ZN5XCRVQ3ZCWNYL7D5JRMWLCMEN5GNLGHYDZXVVYBC4QIQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.381337678 (2h46m10.162271056s) [noise=true , address=77.90.31.16:4735 , piece_id=YADS3OVIRUKMYWHURDC53FA2Y4KGZFPXMKGDQFYBEHIWPBDJQWQA, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.381773217 (2h46m10.162706595s) [noise=true , address=45.91.116.10:4082 , piece_id=GOA4OFIHC7V4VSOSXQVJFNBNF5LDFENYV3VV4DTOAZUTWZYFDIEQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.382058728 (2h46m10.162992093s) [noise=true , address=45.91.116.9:4263 , piece_id=SHN7ZV6VQJ2FB26R3BIRO7O3YHWME7W4WZY4I22C7J4ALNRTZADQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:54.406387202 (2h46m10.187320571s) [noise=true , address=45.91.116.14:4190 , piece_id=DLWQJSE64S5KGO6LR4NCY6K3FS5FPSFGV5AMLJQ6N33Z6VDQM5UQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:37:55.938746597 (2h46m11.719679973s) [noise=true , address=77.90.31.17:4772 , piece_id=I4DSQONAAV4ILKJTSESSHAVRZ3GFG7STRCVLNHLCYSCP3X5S2UPQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: <nil> Jul 27 00:38:24.189189581 (2h46m39.970122947s) [noise=true , address=104.36.82.7:4909 , piece_id=VBLVOC6DNKP6I5URHWF5IUOXWA7CEPYPFETOF47JSHPCCET7VAEQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:24.189252745 (2h46m39.97018611s) [noise=true , address=77.90.31.15:4699 , piece_id=N6NQF3AOROQJFFRXUCYNRSHUI7XB5XNRHGSS4SWFPIVSUUNJAVZQ, seg_pos={0 0}, upload=2554]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:24.189328044 (2h46m39.97026141s) [noise=true , address=104.36.82.4:4803 , piece_id=FFUM4NUBYWJYYUP5RWZJ5BSFNU5RLQIIGGO3MXEYG2TMPDZQYPSA, seg_pos={0 0}, upload=2554]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:24.189376527 (2h46m39.970309893s) [noise=true , address=77.90.31.21:4879 , piece_id=6JOP3SKMWUGQJSFJESPBN35SHIGO4A3XU5SWXUBYHHFO3QFUXE7Q, seg_pos={0 0}, upload=2554]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:24.189425677 (2h46m39.970359046s) [noise=true , address=104.36.82.10:4077 , piece_id=UQWHPVJHCFJWBXTWGFC54RB4N2HIWGTYID4FOJA5Q6MM3PESEM5A, seg_pos={0 0}, upload=2554]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:49848->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:38:24.189446355 (2h46m39.970379732s) [seg_pos={0 0}, upload=2554]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:38:24.223666382 (2h46m40.004599762s) [upload=2554]: Done uploading segments... Jul 27 00:38:24.223707305 (2h46m40.004640686s) [upload=2554]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.037761438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 193.868599ms FAIL (2025-07-27 00:38:20 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 193.904773ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X...XX...............................X...............X..] duration: 30.464027955s FAIL (2025-07-27 00:38:10 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.464053538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X......X.XX....X...X.................XX....X......XX.XXXX...] duration: 1.941315999s FAIL (2025-07-27 00:37:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.941345054s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029120071s FAIL (2025-07-27 00:37:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ing piece... Jul 27 00:36:43.767932742 (2h44m59.548866121s) [noise=true , address=104.36.82.42:4821 , piece_id=UYVX2TUJ7WV5IYW2D357U7ZRQZTU5ZXRMWFOQNBSCHZHFWJHJWPA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768026314 (2h44m59.548959693s) [noise=true , address=79.127.201.214:4321 , piece_id=ITWO3CXXN5NDC7PG6XEYINFLRQ6SGQDS5MM5MLCG35CHS6XOWT5Q, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768368082 (2h44m59.549301459s) [noise=true , address=104.36.82.47:4522 , piece_id=TYMDJXNZRAFTXNRNNBVA2KJ65ZZS5RGBAC7BDPXM2HSJQJ364T3A, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768435991 (2h44m59.54936937s) [noise=true , address=104.36.82.4:4826 , piece_id=GOLCJ7JSJ6OCQUKWPM6F5EV7PGY2HNBSP2DLBXGITRUXYPW6KZNQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768699993 (2h44m59.549633374s) [noise=true , address=45.91.116.11:4130 , piece_id=GENZ6MSX2L4VM5LXHE27PEKIB76GIEQK2UYHKKB37QBAW42PHUEA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768842307 (2h44m59.549775676s) [noise=true , address=79.127.205.235:4687 , piece_id=WJX7AZPSMRF7WUM3LCEDKQJUC6L5TSCSHYAIG75RFRDDLFTQO4YQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768995272 (2h44m59.549928654s) [noise=true , address=77.90.31.15:4720 , piece_id=BJQZAPN2VQGI4UZBODREWLRZB57ZCK37DZYROY6GLNDCBAQEUUZQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.769273110 (2h44m59.550206485s) [noise=true , address=104.36.82.46:4515 , piece_id=R7ZZDKNGNMQ2BZWDJUUNXUFYXGTYMR427DZ7ABW6SVKBPDJDJXSQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.769555178 (2h44m59.550488553s) [noise=true , address=79.127.220.101:4041 , piece_id=USHOT7M2CM7GLICKIQBKB7R2VFQQ7OK5PHG2FS575EL2HHKRRQ3Q, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.769851605 (2h44m59.550784991s) [noise=true , address=64.49.13.137:4502 , piece_id=CDW7WFPA6NBYILJYFUVG3X5SCHRX364RAF7KIRRA62WBONXCXUJQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.770183087 (2h44m59.551116465s) [noise=true , address=45.91.116.10:4093 , piece_id=6HKNOJ3ZJL3YONQYAGPPTDZ64L7YO5G6FBWGEFAORNWG5NHXGYVA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.770520905 (2h44m59.551454283s) [noise=true , address=45.91.116.9:4277 , piece_id=S36THBV2H7SULXMWK6TDJJTLV2K2I2GJGRR7KBVSFQJCKHTRDJ3Q, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.770886534 (2h44m59.551819914s) [noise=true , address=77.90.31.10:4523 , piece_id=WO5HKLFIJ6JDWMGXDSYEGRHP45KEHOOW5ZEMCI66PUTLAIWOOJAA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.771105024 (2h44m59.552038406s) [noise=true , address=104.36.82.56:4231 , piece_id=XGDAWE76KCVZ5FML6ASX4SVQCE5FM2UNARDPPEO3RDLWS7CJBGLA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.771498289 (2h44m59.552431791s) [noise=true , address=104.36.82.7:4926 , piece_id=OKARRYY5ZUAZFQ7QJLFYJTNMQA7VAIXMXZADI6ZMZIIWCZHNJQWA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.771990555 (2h44m59.552923936s) [noise=true , address=79.127.205.234:4612 , piece_id=KFJ7XADVEY64W2M74O2FSHS6HV7GBRLUQENA6MIJXLNS5OUTP6DQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.772401408 (2h44m59.553334794s) [noise=true , address=79.127.201.215:4349 , piece_id=RYGHCEURYGLJKWWNSRGIGZSRNBTE55NP335CGMPLLETIKCMBCCMQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.871311119 (2h44m59.652244485s) [upload=2535]: Next returned nil segment Jul 27 00:36:43.871346890 (2h44m59.652280273s) [upload=2535]: Waiting for error group managing segments... Jul 27 00:36:44.351148542 (2h45m0.132081922s) [noise=true , address=79.127.205.234:4612 , piece_id=KFJ7XADVEY64W2M74O2FSHS6HV7GBRLUQENA6MIJXLNS5OUTP6DQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:44.386342709 (2h45m0.167276094s) [noise=true , address=79.127.205.235:4687 , piece_id=WJX7AZPSMRF7WUM3LCEDKQJUC6L5TSCSHYAIG75RFRDDLFTQO4YQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:45.046538362 (2h45m0.827471727s) [noise=true , address=64.49.13.137:4502 , piece_id=CDW7WFPA6NBYILJYFUVG3X5SCHRX364RAF7KIRRA62WBONXCXUJQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:47.724316954 (2h45m3.505250323s) [noise=true , address=79.127.201.214:4321 , piece_id=ITWO3CXXN5NDC7PG6XEYINFLRQ6SGQDS5MM5MLCG35CHS6XOWT5Q, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:49.376750178 (2h45m5.157683547s) [noise=true , address=45.91.116.11:4130 , piece_id=GENZ6MSX2L4VM5LXHE27PEKIB76GIEQK2UYHKKB37QBAW42PHUEA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:36:50.355177364 (2h45m6.136110743s) [noise=true , address=79.127.220.101:4041 , piece_id=USHOT7M2CM7GLICKIQBKB7R2VFQQ7OK5PHG2FS575EL2HHKRRQ3Q, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:51.429978417 (2h45m7.210911787s) [noise=true , address=45.91.116.10:4093 , piece_id=6HKNOJ3ZJL3YONQYAGPPTDZ64L7YO5G6FBWGEFAORNWG5NHXGYVA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:57.236828201 (2h45m13.017761569s) [noise=true , address=45.91.116.9:4277 , piece_id=S36THBV2H7SULXMWK6TDJJTLV2K2I2GJGRR7KBVSFQJCKHTRDJ3Q, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:37:01.658388202 (2h45m17.439321582s) [noise=true , address=79.127.201.215:4349 , piece_id=RYGHCEURYGLJKWWNSRGIGZSRNBTE55NP335CGMPLLETIKCMBCCMQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:37:23.872581318 (2h45m39.653514688s) [noise=true , address=77.90.31.10:4523 , piece_id=WO5HKLFIJ6JDWMGXDSYEGRHP45KEHOOW5ZEMCI66PUTLAIWOOJAA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:37:43.695477686 (2h45m59.476411062s) [noise=true , address=104.36.82.7:4926 , piece_id=OKARRYY5ZUAZFQ7QJLFYJTNMQA7VAIXMXZADI6ZMZIIWCZHNJQWA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: rpc: dial tcp 104.36.82.7:4926: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695508733 (2h45m59.476442109s) [noise=true , address=104.36.82.47:4522 , piece_id=TYMDJXNZRAFTXNRNNBVA2KJ65ZZS5RGBAC7BDPXM2HSJQJ364T3A, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: write tcp 159.223.143.39:54132->104.36.82.47:4522: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695546438 (2h45m59.476479815s) [noise=true , address=104.36.82.15:4277 , piece_id=QT3D4NZQANATTR6RBNY57NDVAKZ257LY7PFNFKGUWT6NLSPUPUFA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: rpc: dial tcp 104.36.82.15:4277: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695603803 (2h45m59.476537169s) [noise=true , address=104.36.82.46:4515 , piece_id=R7ZZDKNGNMQ2BZWDJUUNXUFYXGTYMR427DZ7ABW6SVKBPDJDJXSQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:43322->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695614733 (2h45m59.476548113s) [noise=true , address=104.36.82.56:4231 , piece_id=XGDAWE76KCVZ5FML6ASX4SVQCE5FM2UNARDPPEO3RDLWS7CJBGLA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695669453 (2h45m59.476602838s) [noise=true , address=104.36.82.4:4826 , piece_id=GOLCJ7JSJ6OCQUKWPM6F5EV7PGY2HNBSP2DLBXGITRUXYPW6KZNQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548, address:104.36.82.4:4826): protocol: rpc: dial tcp 104.36.82.4:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695807449 (2h45m59.476740815s) [noise=true , address=77.90.31.15:4720 , piece_id=BJQZAPN2VQGI4UZBODREWLRZB57ZCK37DZYROY6GLNDCBAQEUUZQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 159.223.143.39:52870->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695902245 (2h45m59.476835626s) [noise=true , address=104.36.82.42:4821 , piece_id=UYVX2TUJ7WV5IYW2D357U7ZRQZTU5ZXRMWFOQNBSCHZHFWJHJWPA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: rpc: dial tcp 104.36.82.42:4821: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:37:43.695961267 (2h45m59.47689465s) [seg_pos={0 0}, upload=2535]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:37:43.721488514 (2h45m59.502421881s) [upload=2535]: Done uploading segments... Jul 27 00:37:43.721531714 (2h45m59.502465083s) [upload=2535]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.029140824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 248.958156ms FAIL (2025-07-27 00:37:39 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 248.984515ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X...XX...............................X...............X.] duration: 51.197774262s FAIL (2025-07-27 00:37:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.197786566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXX] duration: 30.046126676s FAIL (2025-07-27 00:37:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.046172758s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..X] duration: 30.026916589s FAIL (2025-07-27 00:37:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== conn closed: op Jul 27 00:35:58.912470783 (2h44m14.693404168s) [seg_pos={0 0}, upload=2525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:35:58.912556900 (2h44m14.693490281s) [upload=2525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:35:58.912562580 (2h44m14.693495961s) [upload=2525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:35:58.959169793 (2h44m14.740103173s) [upload=2525]: Done uploading segments... Jul 27 00:35:58.959189338 (2h44m14.740122714s) [upload=2525]: Upload finished. err: <nil> Jul 27 00:36:49.246831627 (2h45m5.027765004s) [upload=2538]: Starting upload Jul 27 00:36:49.246924215 (2h45m5.027857599s) [upload=2538]: Done starting upload Jul 27 00:36:49.247146644 (2h45m5.028080026s) [upload=2538]: Uploading segments... Jul 27 00:36:49.247418538 (2h45m5.028351922s) [upload=2538]: Got next segment. Inline: false Jul 27 00:36:49.247493810 (2h45m5.028427181s) [upload=2538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:36:49.247500601 (2h45m5.028433979s) [upload=2538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:36:49.361431565 (2h45m5.142364941s) [seg_pos={0 0}, upload=2538]: Begin upload segment... Jul 27 00:36:49.361486677 (2h45m5.142420053s) [seg_pos={0 0}, upload=2538]: Done begin upload segment. Jul 27 00:36:49.361510789 (2h45m5.142444172s) [noise=true , address=45.91.116.6:4974 , piece_id=XONYL2PYKZUMEKZE4W7WRTV6IVUYFPGBSPSBAFMTI6UH2WTRM2VA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.361525521 (2h45m5.142458909s) [noise=true , address=104.36.82.8:4972 , piece_id=UWC46P6MZRKO336XASYCYE4DNL2EL2GOPTVVFRQHYQHPWUKUHTGA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.361990020 (2h45m5.142923422s) [noise=true , address=77.90.31.11:4554 , piece_id=4TDYCFCSOHMP7VXXX7PXR2PI5CBXIKT5TYRNKLXZ4FHJ7CX7YA3A, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.362114146 (2h45m5.143047535s) [noise=true , address=104.36.82.9:4007 , piece_id=LZBMPOBSL44MAUVUNGRQ5GIBCL6TXMDRHPY73MCNLITKGECE52UA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.362522374 (2h45m5.143455752s) [noise=true , address=77.90.31.10:4517 , piece_id=W36AMIHXFO33E2D5QNFJVTCQXPJNUX5GALTSLYLDMHW3D574UHAQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.362623719 (2h45m5.143557098s) [noise=true , address=45.91.116.5:4297 , piece_id=ZCLLHQOPS3QSUPXBUW5HM2XHCMXRIYADMVFKNB6Q35X5V64R3GIQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.363044271 (2h45m5.143977648s) [noise=true , address=77.90.31.15:4698 , piece_id=R7BVEFUKRNJARZVBW3WGQVE25STNEFPWUXGINXDEIVWTHLNTJX5Q, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.363105102 (2h45m5.144038501s) [noise=true , address=45.91.116.4:4937 , piece_id=B2HCWW4ICKF72LKQ62NUTBUPG4CN6IGTOUWNHA34FCQYUP5O2KTQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.363557375 (2h45m5.144490758s) [noise=true , address=45.91.116.9:4261 , piece_id=GGERWB3IFPNWJ7KME3OF7GKKBJPEFCB62XXV2OOAP4FZW6TYWK6Q, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.364143926 (2h45m5.145077308s) [noise=true , address=77.90.31.14:4661 , piece_id=4UUM63CXD6GGHH2PU3EXEY3576PPE6B7YNA4G6HRNLDD3R7DMSXA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.364549548 (2h45m5.145482924s) [noise=true , address=104.36.82.5:4840 , piece_id=EWHHPZZG7NHPRMEBQOQCDC3A46HX4ZHWR5HO4J4GM2KSRUXAAOFA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.364991231 (2h45m5.145924614s) [noise=true , address=104.36.82.8:4971 , piece_id=IBJI6TBLB5O5C3NSAFCFOJHJN25JHVT47QRLDFNKHSUZSBQXJUSA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.365424115 (2h45m5.146357502s) [noise=true , address=77.90.31.17:4771 , piece_id=3BV5T2W777IZB7PPU5Z737MRKKX2XMSAVAC4HJPXU5MPU2B72RRQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.365821330 (2h45m5.146754714s) [noise=true , address=45.91.116.10:4081 , piece_id=GEGWUVKULKCNRRZW4OYWG42ST3HHVKGZNFJLIDFRTSC6R3RF3JTQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.366327018 (2h45m5.147260405s) [noise=true , address=45.91.116.7:4011 , piece_id=MGHVYRPMHVELCN7MCVS5YGJTB46EE56GZTNJBB5UJ4ANIRSI3FUQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.366885702 (2h45m5.147819094s) [noise=true , address=104.36.82.5:4839 , piece_id=EZIZKT3FTJW6YX4OKQE5JUOUPO6NFN3INKN7UZYZ4RDFJJF376YQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.486871844 (2h45m5.267805231s) [upload=2538]: Next returned nil segment Jul 27 00:36:49.486894239 (2h45m5.267827627s) [upload=2538]: Waiting for error group managing segments... Jul 27 00:36:49.491705522 (2h45m5.272638913s) [noise=true , address=77.90.31.17:4771 , piece_id=3BV5T2W777IZB7PPU5Z737MRKKX2XMSAVAC4HJPXU5MPU2B72RRQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.492059404 (2h45m5.272992785s) [noise=true , address=45.91.116.7:4011 , piece_id=MGHVYRPMHVELCN7MCVS5YGJTB46EE56GZTNJBB5UJ4ANIRSI3FUQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.492530690 (2h45m5.273464059s) [noise=true , address=45.91.116.4:4937 , piece_id=B2HCWW4ICKF72LKQ62NUTBUPG4CN6IGTOUWNHA34FCQYUP5O2KTQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.492789433 (2h45m5.273722812s) [noise=true , address=45.91.116.9:4261 , piece_id=GGERWB3IFPNWJ7KME3OF7GKKBJPEFCB62XXV2OOAP4FZW6TYWK6Q, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.492936408 (2h45m5.273869793s) [noise=true , address=45.91.116.6:4974 , piece_id=XONYL2PYKZUMEKZE4W7WRTV6IVUYFPGBSPSBAFMTI6UH2WTRM2VA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.493417226 (2h45m5.274350598s) [noise=true , address=45.91.116.10:4081 , piece_id=GEGWUVKULKCNRRZW4OYWG42ST3HHVKGZNFJLIDFRTSC6R3RF3JTQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.493578680 (2h45m5.27451206s) [noise=true , address=77.90.31.14:4661 , piece_id=4UUM63CXD6GGHH2PU3EXEY3576PPE6B7YNA4G6HRNLDD3R7DMSXA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.493775828 (2h45m5.274709194s) [noise=true , address=45.91.116.5:4297 , piece_id=ZCLLHQOPS3QSUPXBUW5HM2XHCMXRIYADMVFKNB6Q35X5V64R3GIQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.496148643 (2h45m5.277082022s) [noise=true , address=77.90.31.11:4554 , piece_id=4TDYCFCSOHMP7VXXX7PXR2PI5CBXIKT5TYRNKLXZ4FHJ7CX7YA3A, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:51.476997991 (2h45m7.25793136s) [noise=true , address=77.90.31.10:4517 , piece_id=W36AMIHXFO33E2D5QNFJVTCQXPJNUX5GALTSLYLDMHW3D574UHAQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:37:19.217501107 (2h45m34.998434494s) [noise=true , address=104.36.82.5:4839 , piece_id=EZIZKT3FTJW6YX4OKQE5JUOUPO6NFN3INKN7UZYZ4RDFJJF376YQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:50572->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217623677 (2h45m34.998557053s) [noise=true , address=104.36.82.8:4972 , piece_id=UWC46P6MZRKO336XASYCYE4DNL2EL2GOPTVVFRQHYQHPWUKUHTGA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217699165 (2h45m34.998632552s) [noise=true , address=104.36.82.9:4007 , piece_id=LZBMPOBSL44MAUVUNGRQ5GIBCL6TXMDRHPY73MCNLITKGECE52UA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217742574 (2h45m34.998675953s) [noise=true , address=104.36.82.5:4840 , piece_id=EWHHPZZG7NHPRMEBQOQCDC3A46HX4ZHWR5HO4J4GM2KSRUXAAOFA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217791564 (2h45m34.998724951s) [noise=true , address=104.36.82.8:4971 , piece_id=IBJI6TBLB5O5C3NSAFCFOJHJN25JHVT47QRLDFNKHSUZSBQXJUSA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217838532 (2h45m34.998771913s) [noise=true , address=77.90.31.15:4698 , piece_id=R7BVEFUKRNJARZVBW3WGQVE25STNEFPWUXGINXDEIVWTHLNTJX5Q, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 159.223.143.39:49862->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217865608 (2h45m34.998798994s) [seg_pos={0 0}, upload=2538]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:37:19.241912345 (2h45m35.022845714s) [upload=2538]: Done uploading segments... Jul 27 00:37:19.241938238 (2h45m35.022871613s) [upload=2538]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.026943602s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X......X.XX....X...X.................XX....X......XX.XXXX..] duration: 16.371995472s FAIL (2025-07-27 00:36:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.372025045s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXX] duration: 30.04092315s FAIL (2025-07-27 00:36:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.040957494s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X...XX...............................X...............X] duration: 1m0.001046404s FAIL (2025-07-27 00:36:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001065017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030475186s FAIL (2025-07-27 00:36:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 633s) [noise=true , address=104.36.82.54:4168 , piece_id=GHOKLEOAY3CXMFBWG4ZPJWWO7KHYHMXSDJ3HLB5D5VHMBVOOLI2Q, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.769593367 (2h43m59.550526753s) [noise=true , address=79.127.219.38:4006 , piece_id=JRIFMNH7U5U5HRT2OLGGIPRNYL6UNV2XEPJE7PF3YOIJMTWIZ6DA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770020956 (2h43m59.550954342s) [noise=true , address=79.127.205.246:4277 , piece_id=2K5F6JEVR4Z3PQZAKHXKESAES7HPTBJVPMDURJ7OKKVHQKTQY4KQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770106421 (2h43m59.551039806s) [noise=true , address=77.90.31.17:4801 , piece_id=B4QBFVW6N6SZXZL4R3FPI3KT6HESRE6XUHILSGL6RHAIRBQEVHQQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770435664 (2h43m59.55136905s) [noise=true , address=79.127.205.245:4243 , piece_id=NCKWUZ53ESF5VBR2YUKCNKYXIMRKWMNBFG5UUYOIENTP6IP2IM7Q, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770525511 (2h43m59.551458898s) [noise=true , address=104.36.82.15:4277 , piece_id=7NW6ZCVVR7ZR3PQ65CBIZENJJM3HRL3GMHGQOHL3UNYRXQP37BDA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770823156 (2h43m59.551756543s) [noise=true , address=104.36.82.56:4252 , piece_id=R43YDGDQEUTT4PMDBBA6LHDXUZR4YCBKR43OCD3JEPQI65LQZTZQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770941550 (2h43m59.551874934s) [noise=true , address=104.36.82.44:4886 , piece_id=2AS3FEBOZ2EGVBG3EACVXFKOVE4MDY2CQI4HLPHAQWVMH35DHUPQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.771276510 (2h43m59.552209895s) [noise=true , address=104.36.82.46:4502 , piece_id=Q647ENMCYF7F4JWE4YCRY3VGLXOEQ5ZWQJ5CH6EDEDM2KZ6DLD6Q, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.771415114 (2h43m59.552348499s) [noise=true , address=79.127.205.234:4607 , piece_id=AAUUX32NMVEYZMCE4BJ2SBY46WDQQFEYXP62NDVTHTVTYO4LELEA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.771699733 (2h43m59.552633121s) [noise=true , address=79.127.205.235:4686 , piece_id=ZDUFNQ2MP6JQUW2NUX76TZWSEK62PV5UY5CHY4DWW5YWMYGMAMEA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.772079512 (2h43m59.553012897s) [noise=true , address=104.36.82.53:4151 , piece_id=OBTWN6PCKL4RUGRM4OBCLDLAAULA2RPDG73I2DHMLKQCXOSY3GCQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.772271192 (2h43m59.553204578s) [noise=true , address=79.127.201.215:4355 , piece_id=UOAWKOZ4BIMVDSI4PRGSRIUKFUMXDJQO5BLUVERCM7V2BWHLXGVA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.772534884 (2h43m59.553468269s) [noise=true , address=79.127.205.233:4653 , piece_id=IKCXDR4LAO6HKKBYI77S4VVNPYDIKMGPZGEAQJ3CLKR23LCC7ZHA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.772649281 (2h43m59.553582663s) [noise=true , address=77.90.31.11:4569 , piece_id=WCKYUPNVWSAWPVQI5P4W5CGR3UDCUZ2ITCOR7KWLL2IQVTSRFMMA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.773121589 (2h43m59.554054974s) [noise=true , address=104.36.82.47:4527 , piece_id=K2QVZZMUS4YDQV3XVG7CDESSCDPRIDLDVQOOV33UGVE5JMHE2V6A, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.774109035 (2h43m59.555042417s) [noise=true , address=156.146.43.225:4461 , piece_id=DZF66PYI7MHHJ4WGTOU6GTTFNLSZN7X2HNRZPSIW7PIKNWG6FIPQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.774657740 (2h43m59.555591119s) [noise=true , address=45.91.116.14:4206 , piece_id=3MTQ66Z5BO6E6COJRC6W3S5FXWWQHFTCPA6HUSDZ6LFZOFMQGTYA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.868100219 (2h43m59.649033588s) [upload=2521]: Next returned nil segment Jul 27 00:35:43.868116959 (2h43m59.649050323s) [upload=2521]: Waiting for error group managing segments... Jul 27 00:35:43.872695418 (2h43m59.653628785s) [noise=true , address=79.127.219.38:4006 , piece_id=JRIFMNH7U5U5HRT2OLGGIPRNYL6UNV2XEPJE7PF3YOIJMTWIZ6DA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:43.906564137 (2h43m59.687497521s) [noise=true , address=77.90.31.11:4569 , piece_id=WCKYUPNVWSAWPVQI5P4W5CGR3UDCUZ2ITCOR7KWLL2IQVTSRFMMA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:43.964769076 (2h43m59.745702445s) [noise=true , address=77.90.31.17:4801 , piece_id=B4QBFVW6N6SZXZL4R3FPI3KT6HESRE6XUHILSGL6RHAIRBQEVHQQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.069539990 (2h43m59.850473356s) [noise=true , address=156.146.43.225:4461 , piece_id=DZF66PYI7MHHJ4WGTOU6GTTFNLSZN7X2HNRZPSIW7PIKNWG6FIPQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.353630136 (2h44m0.134563501s) [noise=true , address=79.127.205.234:4607 , piece_id=AAUUX32NMVEYZMCE4BJ2SBY46WDQQFEYXP62NDVTHTVTYO4LELEA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.393257826 (2h44m0.174191208s) [noise=true , address=79.127.205.235:4686 , piece_id=ZDUFNQ2MP6JQUW2NUX76TZWSEK62PV5UY5CHY4DWW5YWMYGMAMEA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.447459336 (2h44m0.228392705s) [noise=true , address=79.127.205.246:4277 , piece_id=2K5F6JEVR4Z3PQZAKHXKESAES7HPTBJVPMDURJ7OKKVHQKTQY4KQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.466280924 (2h44m0.247214293s) [noise=true , address=79.127.205.245:4243 , piece_id=NCKWUZ53ESF5VBR2YUKCNKYXIMRKWMNBFG5UUYOIENTP6IP2IM7Q, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.601309780 (2h44m0.382243149s) [noise=true , address=79.127.205.233:4653 , piece_id=IKCXDR4LAO6HKKBYI77S4VVNPYDIKMGPZGEAQJ3CLKR23LCC7ZHA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:46.698944176 (2h44m2.479877562s) [noise=true , address=45.91.116.14:4206 , piece_id=3MTQ66Z5BO6E6COJRC6W3S5FXWWQHFTCPA6HUSDZ6LFZOFMQGTYA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 27 00:36:02.107058706 (2h44m17.887992087s) [noise=true , address=79.127.201.215:4355 , piece_id=UOAWKOZ4BIMVDSI4PRGSRIUKFUMXDJQO5BLUVERCM7V2BWHLXGVA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:36:43.608681031 (2h44m59.389614416s) [noise=true , address=104.36.82.47:4527 , piece_id=K2QVZZMUS4YDQV3XVG7CDESSCDPRIDLDVQOOV33UGVE5JMHE2V6A, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: rpc: dial tcp 104.36.82.47:4527: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608766466 (2h44m59.389699847s) [noise=true , address=104.36.82.15:4277 , piece_id=7NW6ZCVVR7ZR3PQ65CBIZENJJM3HRL3GMHGQOHL3UNYRXQP37BDA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: write tcp 159.223.143.39:43806->104.36.82.15:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608831139 (2h44m59.389764524s) [noise=true , address=104.36.82.56:4252 , piece_id=R43YDGDQEUTT4PMDBBA6LHDXUZR4YCBKR43OCD3JEPQI65LQZTZQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:1UDFNiqXb4riRuewJDPrKFykVWLiryNKqgkH6uCZszz6Sr8naB, address:104.36.82.56:4252): protocol: write tcp 159.223.143.39:56482->104.36.82.56:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608878390 (2h44m59.389811753s) [noise=true , address=104.36.82.53:4151 , piece_id=OBTWN6PCKL4RUGRM4OBCLDLAAULA2RPDG73I2DHMLKQCXOSY3GCQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:196fMe3fbxKaPgNs1zQx7Kg3Rh68s6Gtp7mbu3hn654WCbZctx, address:104.36.82.53:4151): protocol: write tcp 159.223.143.39:52304->104.36.82.53:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608944475 (2h44m59.389877852s) [noise=true , address=104.36.82.44:4886 , piece_id=2AS3FEBOZ2EGVBG3EACVXFKOVE4MDY2CQI4HLPHAQWVMH35DHUPQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 159.223.143.39:43574->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608977323 (2h44m59.389910699s) [noise=true , address=104.36.82.54:4168 , piece_id=GHOKLEOAY3CXMFBWG4ZPJWWO7KHYHMXSDJ3HLB5D5VHMBVOOLI2Q, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: rpc: dial tcp 104.36.82.54:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.609024323 (2h44m59.38995769s) [noise=true , address=104.36.82.46:4502 , piece_id=Q647ENMCYF7F4JWE4YCRY3VGLXOEQ5ZWQJ5CH6EDEDM2KZ6DLD6Q, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: write tcp 159.223.143.39:36674->104.36.82.46:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.609061434 (2h44m59.38999482s) [seg_pos={0 0}, upload=2521]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:36:43.634116116 (2h44m59.415049486s) [upload=2521]: Done uploading segments... Jul 27 00:36:43.634148701 (2h44m59.415082073s) [upload=2521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.030499246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 190.600788ms FAIL (2025-07-27 00:36:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 190.627062ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)