Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...................................................XX] duration: 30.046549474s FAIL (2025-07-26 23:08:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 97685931 (1h16m33.578619318s) [upload=1181]: Starting upload Jul 26 23:08:17.797842115 (1h16m33.578775501s) [upload=1181]: Done starting upload Jul 26 23:08:17.798039081 (1h16m33.578972452s) [upload=1181]: Uploading segments... Jul 26 23:08:17.798473555 (1h16m33.57940692s) [upload=1181]: Got next segment. Inline: false Jul 26 23:08:17.798565517 (1h16m33.579498897s) [upload=1181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:08:17.798572280 (1h16m33.579505656s) [upload=1181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:08:17.824915523 (1h16m33.6058489s) [seg_pos={0 0}, upload=1181]: Begin upload segment... Jul 26 23:08:17.824969912 (1h16m33.605903288s) [seg_pos={0 0}, upload=1181]: Done begin upload segment. Jul 26 23:08:17.824997041 (1h16m33.60593042s) [noise=true , address=45.91.116.7:4010 , piece_id=N66J3MYD4QBVDMDOP3NNTQVGMSJCTWXHY34NS3H7JRE6T3SBDIWQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.825563272 (1h16m33.606496661s) [noise=true , address=104.36.82.8:4969 , piece_id=L5F6DW4B2W5EQW7THR4O53SMBQAPNTWW45BD6KV6VB45IWMA6ZDA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.826673992 (1h16m33.607607379s) [noise=true , address=45.91.116.4:4938 , piece_id=TV2GG5UHK3E4HIUKVBCCZGKPBKWBCCS6JOZBLOG3EIXQG6GMM3NA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.827185284 (1h16m33.608118711s) [noise=true , address=104.36.82.14:4223 , piece_id=P5L2RLUSGY375RHA7HNLKEVOF5QLCRIHY6F57DJI5MSQFVQ7VGXQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.827636178 (1h16m33.608569545s) [noise=true , address=104.36.82.15:4259 , piece_id=OPTCVANC4E5OWF35VIS7NMTWNVYC4B6MYYSAIIBTUFSAPXWBAUWA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.827966744 (1h16m33.608900126s) [noise=true , address=104.36.82.4:4804 , piece_id=ET4V42VZ3XOX6X2LS4F2VMQOKU5ATGNMESSI2WLHFWQQ42A4OBJA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.828361630 (1h16m33.609295007s) [noise=true , address=104.36.82.9:4007 , piece_id=IN6MWO2P2A7M64KSBXZALRKCSP46OLFDUKV73WCQGMGZFSXG6QVA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.828907321 (1h16m33.609840707s) [noise=true , address=104.36.82.2:4732 , piece_id=EL5YIXTMUJEKXGFTJ5ZHTQZUJW4ELYSBQ4YDDCMSTSOTCXVM37MQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.829116385 (1h16m33.610049765s) [noise=true , address=45.91.116.11:4118 , piece_id=LFGDECRUIOJLRPUO5NWOVQNMG6SKQVUT4RY6QPBPGKOVHROBVPQQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.829419025 (1h16m33.610352405s) [noise=true , address=104.36.82.3:4765 , piece_id=OIFXWMKVREKJVSRKY6KQRVKIEQRIOSQBT7AR72T67LNUECD4ALCA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.829504495 (1h16m33.610437883s) [noise=true , address=77.90.31.14:4663 , piece_id=R4NQENA2VJEDN3HV6X4LYWELNHWEI5TGJFU3GVCCRZ5LKEMUZ3BQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.829923294 (1h16m33.610856675s) [noise=true , address=45.91.116.8:4048 , piece_id=SPTOI42AXO5S666EWMRAQMEPQ4CLLMDV7SP2FW5WISCFLIHCQHNQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.830391997 (1h16m33.611325385s) [noise=true , address=45.91.116.4:4940 , piece_id=LWV5TAY5XETIWZRO3MQZEVH7HZUJKQ52QJ5T43MGNYARSHCUOLHQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.830655447 (1h16m33.611588826s) [noise=true , address=104.36.82.8:4972 , piece_id=32QBA4R5S4APFXWMJLKEZCDVMOIJ4LXOZATGGQUVBKAXEGWVFY6A, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.831648357 (1h16m33.61258175s) [noise=true , address=104.36.82.6:4876 , piece_id=6FIJW7ZPY2BY2L374UUXQ563SMSWFJMNB4VB5X5WNZ55MAAU7WZA, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.832405120 (1h16m33.613338533s) [noise=true , address=45.91.116.12:4227 , piece_id=PEBKEZBE2KV5IGCTGI222OCYI6NO7UPVJX66V3JZVOXL32XCVNMQ, seg_pos={0 0}, upload=1181]: Uploading piece... Jul 26 23:08:17.915119182 (1h16m33.696052559s) [upload=1181]: Next returned nil segment Jul 26 23:08:17.915141507 (1h16m33.696074892s) [upload=1181]: Waiting for error group managing segments... Jul 26 23:08:17.919860312 (1h16m33.700793696s) [noise=true , address=45.91.116.7:4010 , piece_id=N66J3MYD4QBVDMDOP3NNTQVGMSJCTWXHY34NS3H7JRE6T3SBDIWQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.921371973 (1h16m33.702305354s) [noise=true , address=45.91.116.11:4118 , piece_id=LFGDECRUIOJLRPUO5NWOVQNMG6SKQVUT4RY6QPBPGKOVHROBVPQQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.921938398 (1h16m33.702871761s) [noise=true , address=45.91.116.8:4048 , piece_id=SPTOI42AXO5S666EWMRAQMEPQ4CLLMDV7SP2FW5WISCFLIHCQHNQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.922448973 (1h16m33.703382335s) [noise=true , address=45.91.116.4:4940 , piece_id=LWV5TAY5XETIWZRO3MQZEVH7HZUJKQ52QJ5T43MGNYARSHCUOLHQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.923088658 (1h16m33.70402202s) [noise=true , address=45.91.116.4:4938 , piece_id=TV2GG5UHK3E4HIUKVBCCZGKPBKWBCCS6JOZBLOG3EIXQG6GMM3NA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.923644261 (1h16m33.704577647s) [noise=true , address=45.91.116.12:4227 , piece_id=PEBKEZBE2KV5IGCTGI222OCYI6NO7UPVJX66V3JZVOXL32XCVNMQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:17.929203579 (1h16m33.710136962s) [noise=true , address=77.90.31.14:4663 , piece_id=R4NQENA2VJEDN3HV6X4LYWELNHWEI5TGJFU3GVCCRZ5LKEMUZ3BQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: <nil> Jul 26 23:08:47.760958504 (1h17m3.541891893s) [noise=true , address=104.36.82.14:4223 , piece_id=P5L2RLUSGY375RHA7HNLKEVOF5QLCRIHY6F57DJI5MSQFVQ7VGXQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.761069173 (1h17m3.542002537s) [noise=true , address=104.36.82.15:4259 , piece_id=OPTCVANC4E5OWF35VIS7NMTWNVYC4B6MYYSAIIBTUFSAPXWBAUWA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.761152042 (1h17m3.542085416s) [noise=true , address=104.36.82.3:4765 , piece_id=OIFXWMKVREKJVSRKY6KQRVKIEQRIOSQBT7AR72T67LNUECD4ALCA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.761324310 (1h17m3.542257691s) [noise=true , address=104.36.82.6:4876 , piece_id=6FIJW7ZPY2BY2L374UUXQ563SMSWFJMNB4VB5X5WNZ55MAAU7WZA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:08:47.763634370 (1h17m3.544567761s) [noise=true , address=104.36.82.8:4969 , piece_id=L5F6DW4B2W5EQW7THR4O53SMBQAPNTWW45BD6KV6VB45IWMA6ZDA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.763764969 (1h17m3.544698334s) [noise=true , address=104.36.82.9:4007 , piece_id=IN6MWO2P2A7M64KSBXZALRKCSP46OLFDUKV73WCQGMGZFSXG6QVA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.763856186 (1h17m3.544789564s) [noise=true , address=104.36.82.8:4972 , piece_id=32QBA4R5S4APFXWMJLKEZCDVMOIJ4LXOZATGGQUVBKAXEGWVFY6A, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:46250->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.764028239 (1h17m3.544961611s) [noise=true , address=104.36.82.4:4804 , piece_id=ET4V42VZ3XOX6X2LS4F2VMQOKU5ATGNMESSI2WLHFWQQ42A4OBJA, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:47.764270783 (1h17m3.545204146s) [noise=true , address=104.36.82.2:4732 , piece_id=EL5YIXTMUJEKXGFTJ5ZHTQZUJW4ELYSBQ4YDDCMSTSOTCXVM37MQ, seg_pos={0 0}, upload=1181]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:51900->104.36.82.2:4732: use of closed network connection; conn closed: op Jul 26 23:08:47.764301402 (1h17m3.545234793s) [seg_pos={0 0}, upload=1181]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:08:47.798682499 (1h17m3.579615873s) [upload=1181]: Done uploading segments... Jul 26 23:08:47.798712233 (1h17m3.579645833s) [upload=1181]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.046575849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXX] duration: 1m0.038879132s FAIL (2025-07-26 23:08:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 1]: Uploading piece... Jul 26 23:07:40.682030269 (1h15m56.462963648s) [noise=true , address=104.36.82.47:4541 , piece_id=HF3L73Q6NTSUH4HMAGJAVHW4T3UNHB6XBBZWFAW6XJQIY4HKQZBA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.682294597 (1h15m56.463227981s) [noise=true , address=104.36.82.42:4805 , piece_id=2JBJZS5KD2ULEKX2JAHTK466WH5EHH6KDLJJPCGTWX2NTPH3VQSQ, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.682577978 (1h15m56.463511371s) [noise=true , address=104.36.82.3:4784 , piece_id=6AMRXJOLZUUCTD3KSO3COBIMLSCJGMTY3ER573VLFNP6D42C5NUA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.682859361 (1h15m56.463792734s) [noise=true , address=104.36.82.46:4508 , piece_id=2QXA2OCHDHMIYI2FXHWOJLLOLHA2TEHWWTZEZV45QADEN23NXDAQ, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.683313077 (1h15m56.464246464s) [noise=true , address=79.127.205.233:4650 , piece_id=DG2KEGNOEFZUKIGKZ5VPGXMC7YSCSYUSTSXY4DTBV6Q6WEOLKZRQ, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.709755308 (1h15m56.490688696s) [noise=true , address=104.36.82.40:4953 , piece_id=UIUPAX6A7ABU3ZAFQVIIIYGQJCRZE5JOHTDUF4CYOEABN4EMTJQA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.711181562 (1h15m56.49211496s) [noise=true , address=104.36.82.53:4133 , piece_id=4FU4UFMDXNTUQZ2TIMSEA6VPLGPXTFQBO3A7RUYSU2LTXBWHM45A, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.711652969 (1h15m56.492586357s) [noise=true , address=104.36.82.36:4650 , piece_id=4RONA555PXK2FQS5ERFID5MXAEYSS65JGFHK5QRWDLVXDRWPLKDA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.712089569 (1h15m56.493022956s) [noise=true , address=104.36.82.35:4612 , piece_id=DD6YHE77ZZWYSWKX2MAU7OUM6OJP7JPJWYVMNRWZO7O4PSECFHUA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.712510112 (1h15m56.493443504s) [noise=true , address=104.36.82.44:4885 , piece_id=CX7I6JHDTUNXPUSST7UCC6X6PCWZGEDMWIUVJRJLYR2FRUCHGP7A, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.712906845 (1h15m56.493840213s) [noise=true , address=104.36.82.21:4485 , piece_id=FDWUOES5NG5LIZCNKRDW5DRSXMZ4EM53WBW3564N5WBCJPHQ5XFA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.713257134 (1h15m56.494190522s) [noise=true , address=77.90.31.13:4658 , piece_id=F4SXB3B2TV57K2WTI6YVOSYQRKPU2DOZ3PU6SSB2JCB2TJSW3YBQ, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.713609748 (1h15m56.49454313s) [noise=true , address=104.36.82.56:4264 , piece_id=JWOHBFMD3YG2LFHMQQZQ3E7UQ43HPDERKH6MIJNNNM4O3DP43CBA, seg_pos={0 0}, upload=1171]: Uploading piece... Jul 26 23:07:40.815609578 (1h15m56.596542947s) [upload=1171]: Next returned nil segment Jul 26 23:07:40.815625479 (1h15m56.596558848s) [upload=1171]: Waiting for error group managing segments... Jul 26 23:07:40.822002686 (1h15m56.602936055s) [noise=true , address=79.127.219.38:4018 , piece_id=7DKMGNGVS27SKALWMZJ3XE7LMPIRLAZ3MXR2KYSQM6VA2AIA36NQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:40.834523370 (1h15m56.615456751s) [noise=true , address=104.36.82.52:4101 , piece_id=GVNIIJGYFBY2FIJVZX7L5MH4YV3QMVZ4LMTUWHQSDOT6PIEOTLOQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:40.836884698 (1h15m56.617818081s) [noise=true , address=104.36.82.56:4264 , piece_id=JWOHBFMD3YG2LFHMQQZQ3E7UQ43HPDERKH6MIJNNNM4O3DP43CBA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:40.841110406 (1h15m56.622043786s) [noise=true , address=104.36.82.53:4133 , piece_id=4FU4UFMDXNTUQZ2TIMSEA6VPLGPXTFQBO3A7RUYSU2LTXBWHM45A, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:41.323408089 (1h15m57.104341472s) [noise=true , address=79.127.205.233:4650 , piece_id=DG2KEGNOEFZUKIGKZ5VPGXMC7YSCSYUSTSXY4DTBV6Q6WEOLKZRQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:07:47.827730169 (1h16m3.608663546s) [noise=true , address=104.36.82.54:4168 , piece_id=CJBMJASUWS74DKCYLZEYBGWTZ6CBJYY4YOPHNQ2YF4R3GKSMISXA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: <nil> Jul 26 23:08:40.610534953 (1h16m56.391468339s) [noise=true , address=104.36.82.42:4805 , piece_id=2JBJZS5KD2ULEKX2JAHTK466WH5EHH6KDLJJPCGTWX2NTPH3VQSQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1nh6Kih617STw2M8iHVjs4xP1HjjckJHyUxQ5z6ZFRhWwG6RXi, address:104.36.82.42:4805): protocol: rpc: dial tcp 104.36.82.42:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610647891 (1h16m56.391581276s) [noise=true , address=104.36.82.3:4784 , piece_id=6AMRXJOLZUUCTD3KSO3COBIMLSCJGMTY3ER573VLFNP6D42C5NUA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3, address:104.36.82.3:4784): protocol: rpc: dial tcp 104.36.82.3:4784: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610733661 (1h16m56.391667041s) [noise=true , address=104.36.82.41:4785 , piece_id=YP77FRE3L2SRTNKP43PMEXIUR2EG2WJSL3QUA2Z45CXHCFKW66QQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB, address:104.36.82.41:4785): protocol: rpc: dial tcp 104.36.82.41:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610796410 (1h16m56.391729789s) [noise=true , address=104.36.82.40:4953 , piece_id=UIUPAX6A7ABU3ZAFQVIIIYGQJCRZE5JOHTDUF4CYOEABN4EMTJQA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:12dhNW98YkWKWzXBnwmTCvzaw9CtvxRW8ELJnAjHnATFhpM1oKn, address:104.36.82.40:4953): protocol: rpc: dial tcp 104.36.82.40:4953: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610821283 (1h16m56.391754662s) [noise=true , address=104.36.82.47:4541 , piece_id=HF3L73Q6NTSUH4HMAGJAVHW4T3UNHB6XBBZWFAW6XJQIY4HKQZBA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610887841 (1h16m56.39182122s) [noise=true , address=104.36.82.44:4885 , piece_id=CX7I6JHDTUNXPUSST7UCC6X6PCWZGEDMWIUVJRJLYR2FRUCHGP7A, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: rpc: dial tcp 104.36.82.44:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610941206 (1h16m56.391874588s) [noise=true , address=104.36.82.21:4485 , piece_id=FDWUOES5NG5LIZCNKRDW5DRSXMZ4EM53WBW3564N5WBCJPHQ5XFA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1wmohWyFaMkNm8KBAZprmZ6RuiH6UcB8KRNght2HGpV3Ucnnq9, address:104.36.82.21:4485): protocol: write tcp 159.223.143.39:58518->104.36.82.21:4485: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610951572 (1h16m56.39188494s) [noise=true , address=104.36.82.36:4650 , piece_id=4RONA555PXK2FQS5ERFID5MXAEYSS65JGFHK5QRWDLVXDRWPLKDA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1FM1g8LyQ9LJCNjKLxJtTQyDNudVcN4tv3NsxKRTV5gVChjCzp, address:104.36.82.36:4650): protocol: write tcp 159.223.143.39:43804->104.36.82.36:4650: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.610965687 (1h16m56.391899066s) [noise=true , address=104.36.82.22:4538 , piece_id=3TH2UZP6QEODNVFFDSAXZZ7KQEBC2ZSELBXPCOKZKXU45VKDD3YQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:104.36.82.22:4538): protocol: write tcp 159.223.143.39:44462->104.36.82.22:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.611083592 (1h16m56.392016972s) [noise=true , address=77.90.31.13:4658 , piece_id=F4SXB3B2TV57K2WTI6YVOSYQRKPU2DOZ3PU6SSB2JCB2TJSW3YBQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: read tcp 159.223.143.39:57360->77.90.31.13:4658: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.611097271 (1h16m56.392030641s) [noise=true , address=104.36.82.35:4612 , piece_id=DD6YHE77ZZWYSWKX2MAU7OUM6OJP7JPJWYVMNRWZO7O4PSECFHUA, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: rpc: dial tcp 104.36.82.35:4612: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.611123970 (1h16m56.392057349s) [noise=true , address=104.36.82.46:4508 , piece_id=2QXA2OCHDHMIYI2FXHWOJLLOLHA2TEHWWTZEZV45QADEN23NXDAQ, seg_pos={0 0}, upload=1171]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: read tcp 159.223.143.39:38136->104.36.82.46:4508: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:40.611156836 (1h16m56.3920902s) [seg_pos={0 0}, upload=1171]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 23:08:40.644393326 (1h16m56.425326708s) [upload=1171]: Done uploading segments... Jul 26 23:08:40.644441483 (1h16m56.425374862s) [upload=1171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.038903503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.002447769s FAIL (2025-07-26 23:08:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002467159s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................................X] duration: 30.032342542s FAIL (2025-07-26 23:08:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 23:07:46.502507465 (1h16m2.283440843s) [upload=1175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:07:46.502514698 (1h16m2.283448083s) [upload=1175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:07:46.538591482 (1h16m2.319524896s) [seg_pos={0 0}, upload=1175]: Begin upload segment... Jul 26 23:07:46.538666369 (1h16m2.319599753s) [seg_pos={0 0}, upload=1175]: Done begin upload segment. Jul 26 23:07:46.538988284 (1h16m2.319921668s) [noise=true , address=104.36.82.6:4875 , piece_id=MUBYCAFZZNFAWUZDOZZQIKAFQSIZWF7AEEZRF47INQ2EAF66NNZA, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.539079551 (1h16m2.320012938s) [noise=true , address=104.36.82.15:4257 , piece_id=N7UGWSCY4LR5TPC5RED7YDIW5TIZMXPZ35SUWTVPIZTRR6BFMDMQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.539804658 (1h16m2.320738047s) [noise=true , address=104.36.82.2:4729 , piece_id=XXZNOYSYRIIJIR54NTZXL7F7XXYIPCLJY4X6NGV2VLSD43UD2QFQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.541071101 (1h16m2.322004499s) [noise=true , address=104.36.82.14:4223 , piece_id=YQFOFJBGZMDVNGIAU7NR5HWMNN4SXWKKLNRFRYMMW2O6QUGA7SKQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.541194964 (1h16m2.322128347s) [noise=true , address=104.36.82.8:4970 , piece_id=WIZILE4HKCT4EBR7NO6USMQLVH3ANCRKXUIMDSRGK2W6BZNAOJGQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.541735822 (1h16m2.322669208s) [noise=true , address=45.91.116.5:4297 , piece_id=QOL343MXQK3D5PXHWAAXVJJXUVVRDSRXOYUGDXUWA6E6NFW5BW6A, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.542375818 (1h16m2.323309208s) [noise=true , address=45.91.116.4:4938 , piece_id=MVZ2K4WJEN4GNWCO2RXE6XCF7Y7YXWV5UVD6K4PPEIKUMK5JG7PQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.542768507 (1h16m2.323701895s) [noise=true , address=45.91.116.8:4047 , piece_id=AVYWRB5JKMKRVUBM7CONKSFLWEVBMKOCHRV3ERSXNVTPB6VNFAEQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.543380518 (1h16m2.324313908s) [noise=true , address=104.36.82.15:4258 , piece_id=MSSR4RXTG4RQMJR6DWE2IISGHEHURQGIZUJ3TNZW5XX64ZGM3RLA, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.544602822 (1h16m2.325536213s) [noise=true , address=104.36.82.4:4802 , piece_id=WOP737BTCQ3G6NUWIGKRRZR6KYMPHN7WUJ5HOPZTNYHK45K77H7Q, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.545809319 (1h16m2.32674271s) [noise=true , address=104.36.82.16:4293 , piece_id=D3PVF6AD6KLKK6V7TCXC7T4R3ILAZVJVCDREFPVG3SMYJTZYHIKQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.546515257 (1h16m2.327448646s) [noise=true , address=45.91.116.6:4974 , piece_id=66AHC6FRA3L6YS2OIHPAAUQDWNWPR3LXOMWZ2AVTY53QJXD4PIMA, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.548171938 (1h16m2.329105319s) [noise=true , address=77.90.31.14:4663 , piece_id=MAHGERLTWI4DAP27TPQWJ3YRK3A2DA643QG5ZQ5B466SYGIV242A, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.548972775 (1h16m2.329906147s) [noise=true , address=104.36.82.9:4005 , piece_id=26LO3JCYB42R3QGC4ENR7XORRKUW76KHMWRWKOPO2AKPNUT2PAXQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.549556098 (1h16m2.330489479s) [noise=true , address=45.91.116.7:4010 , piece_id=P7GSPVLVGKS6YEMINKYIIA2E6MN3LH2CIMOT5MWTKYO2JJLMBNAA, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.550145176 (1h16m2.331078566s) [noise=true , address=45.91.116.5:4299 , piece_id=J36A2QN72CO4ACOEEZXJ4BHYJAOQ533QYAK5QJC2N2M6ASZ6KJFQ, seg_pos={0 0}, upload=1175]: Uploading piece... Jul 26 23:07:46.677350539 (1h16m2.458283921s) [upload=1175]: Next returned nil segment Jul 26 23:07:46.677455541 (1h16m2.458388927s) [upload=1175]: Waiting for error group managing segments... Jul 26 23:07:46.683203189 (1h16m2.464136574s) [noise=true , address=45.91.116.6:4974 , piece_id=66AHC6FRA3L6YS2OIHPAAUQDWNWPR3LXOMWZ2AVTY53QJXD4PIMA, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.684101593 (1h16m2.465034973s) [noise=true , address=45.91.116.5:4297 , piece_id=QOL343MXQK3D5PXHWAAXVJJXUVVRDSRXOYUGDXUWA6E6NFW5BW6A, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.684245646 (1h16m2.465179013s) [noise=true , address=77.90.31.14:4663 , piece_id=MAHGERLTWI4DAP27TPQWJ3YRK3A2DA643QG5ZQ5B466SYGIV242A, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.684707108 (1h16m2.465640489s) [noise=true , address=45.91.116.7:4010 , piece_id=P7GSPVLVGKS6YEMINKYIIA2E6MN3LH2CIMOT5MWTKYO2JJLMBNAA, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.684965874 (1h16m2.465899243s) [noise=true , address=45.91.116.8:4047 , piece_id=AVYWRB5JKMKRVUBM7CONKSFLWEVBMKOCHRV3ERSXNVTPB6VNFAEQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.686300892 (1h16m2.467234257s) [noise=true , address=45.91.116.5:4299 , piece_id=J36A2QN72CO4ACOEEZXJ4BHYJAOQ533QYAK5QJC2N2M6ASZ6KJFQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:07:46.687185916 (1h16m2.46811928s) [noise=true , address=45.91.116.4:4938 , piece_id=MVZ2K4WJEN4GNWCO2RXE6XCF7Y7YXWV5UVD6K4PPEIKUMK5JG7PQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: <nil> Jul 26 23:08:16.468753767 (1h16m32.249687156s) [noise=true , address=104.36.82.2:4729 , piece_id=XXZNOYSYRIIJIR54NTZXL7F7XXYIPCLJY4X6NGV2VLSD43UD2QFQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:38346->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468798397 (1h16m32.249731773s) [noise=true , address=104.36.82.9:4005 , piece_id=26LO3JCYB42R3QGC4ENR7XORRKUW76KHMWRWKOPO2AKPNUT2PAXQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:56398->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468859512 (1h16m32.249792889s) [noise=true , address=104.36.82.4:4802 , piece_id=WOP737BTCQ3G6NUWIGKRRZR6KYMPHN7WUJ5HOPZTNYHK45K77H7Q, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468923799 (1h16m32.249857169s) [noise=true , address=104.36.82.8:4970 , piece_id=WIZILE4HKCT4EBR7NO6USMQLVH3ANCRKXUIMDSRGK2W6BZNAOJGQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 159.223.143.39:39368->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468941115 (1h16m32.249874497s) [noise=true , address=104.36.82.15:4257 , piece_id=N7UGWSCY4LR5TPC5RED7YDIW5TIZMXPZ35SUWTVPIZTRR6BFMDMQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:36836->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.468980078 (1h16m32.249913459s) [noise=true , address=104.36.82.15:4258 , piece_id=MSSR4RXTG4RQMJR6DWE2IISGHEHURQGIZUJ3TNZW5XX64ZGM3RLA, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:52846->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.469051468 (1h16m32.249984846s) [noise=true , address=104.36.82.16:4293 , piece_id=D3PVF6AD6KLKK6V7TCXC7T4R3ILAZVJVCDREFPVG3SMYJTZYHIKQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:38550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.469067759 (1h16m32.250001137s) [noise=true , address=104.36.82.6:4875 , piece_id=MUBYCAFZZNFAWUZDOZZQIKAFQSIZWF7AEEZRF47INQ2EAF66NNZA, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:48866->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.469096507 (1h16m32.250029875s) [noise=true , address=104.36.82.14:4223 , piece_id=YQFOFJBGZMDVNGIAU7NR5HWMNN4SXWKKLNRFRYMMW2O6QUGA7SKQ, seg_pos={0 0}, upload=1175]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:52262->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:16.469153422 (1h16m32.250086957s) [seg_pos={0 0}, upload=1175]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:08:16.495582206 (1h16m32.276515679s) [upload=1175]: Done uploading segments... Jul 26 23:08:16.495628545 (1h16m32.276561947s) [upload=1175]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.032380819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X................X........................X......XX] duration: 30.033439715s FAIL (2025-07-26 23:08:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.033490399s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001421453s FAIL (2025-07-26 23:08:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001448019s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 218.912172ms FAIL (2025-07-26 23:07:44 UTC) orbiter warning: uplink: too many requests Duration: 218.944248ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XX] duration: 1m0.029265261s FAIL (2025-07-26 23:07:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== QTYDCF4B7ITENJ4IHP5CF2GQ5IBTVOA, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.635602680 (1h14m56.416536068s) [noise=true , address=104.36.82.12:4182 , piece_id=7BUAGBJEH6U35GFWZJZ2ALOZDWJY6IHRTVUSQIH5P3WCZ57GFDWQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.635666958 (1h14m56.416600343s) [noise=true , address=104.36.82.4:4835 , piece_id=6ONX3EUIHN3ZV3ZXKLYPMMPPMNKDSGCPCR3I2SQZDJG6ODZIWVAA, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.636221817 (1h14m56.417155202s) [noise=true , address=79.127.205.234:4613 , piece_id=PS4KK6FV2GS6CA5ZUCG37XSWUWKJEXFO4IA6QQ4BBYTDUXIFJTSQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.636282230 (1h14m56.417215617s) [noise=true , address=104.36.82.46:4513 , piece_id=3QXJWIYI2UCJSXY7DYYT36MLNLAKEZ7BFIXQ5PV4PXYG52UYTEAQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.636628901 (1h14m56.417562281s) [noise=true , address=104.36.82.11:4139 , piece_id=VHTHZV5MFUKN5KNKLUQJMM7BZGBJCOALV6SFDWDKDQVSGDV72MJQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.636837528 (1h14m56.417770913s) [noise=true , address=104.36.82.52:4091 , piece_id=A36364UTYE5BWAJLV7EBHMKEA7LXHEYSFPPKXBFGGLKQFRGEBL6Q, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.637048027 (1h14m56.417981418s) [noise=true , address=104.36.82.56:4262 , piece_id=5P4NUWKQMZHUHAEDMS5RFI2B76DSBTVOCE7WUXUXK2YGUXZW47BA, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.637333667 (1h14m56.418267056s) [noise=true , address=79.127.205.235:4683 , piece_id=6YOBQICR5E5MDO3XDFHZSJK35Z7QTBGOYCKIJYV3JQTELC7FGI3A, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.637796008 (1h14m56.418729393s) [noise=true , address=104.36.82.9:4015 , piece_id=XE7TTRTSXBAQQI7G5TPTLXZ3DUOWMQ5JMI6M5R5EBJI76WKSLO2A, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.638243205 (1h14m56.419176594s) [noise=true , address=104.36.82.22:4535 , piece_id=TIEDRQ7T5XR5RIY3YZE46JTWEIZJSNNIGTGGBFX5EY3FC2WU5I4A, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.638639898 (1h14m56.41957327s) [noise=true , address=104.36.82.54:4172 , piece_id=T6PAR63ENE22FBS5OLL7E2MMVYLTJ3LUNFEAF36DRQKQSDAEPGKQ, seg_pos={0 0}, upload=1157]: Uploading piece... Jul 26 23:06:40.822810191 (1h14m56.603743577s) [upload=1157]: Next returned nil segment Jul 26 23:06:40.822832993 (1h14m56.603766385s) [upload=1157]: Waiting for error group managing segments... Jul 26 23:06:40.853264663 (1h14m56.634198058s) [noise=true , address=79.127.220.101:4048 , piece_id=5WHWPJZSQ4JYUPXIEVTHDUGHBD3UGOIXPN7AC2VZOFTORRQPJC2Q, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:06:40.928131201 (1h14m56.709064575s) [noise=true , address=156.146.43.225:4449 , piece_id=LAUVUGX2Y73O4DKUK4DNR7O3H65SOK6LEGGNVKU32XND2BLX4AHQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:06:41.259976183 (1h14m57.040909555s) [noise=true , address=79.127.205.234:4613 , piece_id=PS4KK6FV2GS6CA5ZUCG37XSWUWKJEXFO4IA6QQ4BBYTDUXIFJTSQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:06:41.343659822 (1h14m57.12459319s) [noise=true , address=79.127.205.235:4683 , piece_id=6YOBQICR5E5MDO3XDFHZSJK35Z7QTBGOYCKIJYV3JQTELC7FGI3A, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:07:36.063582207 (1h15m51.844515586s) [noise=true , address=104.36.82.56:4262 , piece_id=5P4NUWKQMZHUHAEDMS5RFI2B76DSBTVOCE7WUXUXK2YGUXZW47BA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: <nil> Jul 26 23:07:40.507730600 (1h15m56.288663977s) [noise=true , address=104.36.82.54:4172 , piece_id=T6PAR63ENE22FBS5OLL7E2MMVYLTJ3LUNFEAF36DRQKQSDAEPGKQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: write tcp 159.223.143.39:44996->104.36.82.54:4172: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.507836646 (1h15m56.288770016s) [noise=true , address=104.36.82.9:4015 , piece_id=XE7TTRTSXBAQQI7G5TPTLXZ3DUOWMQ5JMI6M5R5EBJI76WKSLO2A, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: write tcp 159.223.143.39:54344->104.36.82.9:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.507935623 (1h15m56.288869008s) [noise=true , address=104.36.82.51:4049 , piece_id=PDJJ62LDJVUJXSIBRMK6LQXPSVRX7FCQLNAL6CNJWOVQLSKTMWGQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:104.36.82.51:4049): protocol: write tcp 159.223.143.39:59346->104.36.82.51:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508198389 (1h15m56.289131752s) [noise=true , address=104.36.82.41:4771 , piece_id=2FLO4QF57SARKMTNRCURZMHLYRKGJOMMHW6CBAW2QWCJM7QMOFLA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto, address:104.36.82.41:4771): protocol: rpc: dial tcp 104.36.82.41:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508431420 (1h15m56.289364802s) [noise=true , address=104.36.82.11:4139 , piece_id=VHTHZV5MFUKN5KNKLUQJMM7BZGBJCOALV6SFDWDKDQVSGDV72MJQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: rpc: dial tcp 104.36.82.11:4139: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508498866 (1h15m56.289432229s) [noise=true , address=104.36.82.46:4513 , piece_id=3QXJWIYI2UCJSXY7DYYT36MLNLAKEZ7BFIXQ5PV4PXYG52UYTEAQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: read tcp 159.223.143.39:37300->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508592196 (1h15m56.289525573s) [noise=true , address=104.36.82.47:4517 , piece_id=PPDGZ2NUDEJEJNTHMETWVQAY4RRLFUUTUAYETXOPKHNYSVZRWJGA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508661526 (1h15m56.289594903s) [noise=true , address=104.36.82.49:4978 , piece_id=FDORIP3PHQQXXEXGWTX6Y4MWCOSFFNPTK24ZQX5QQA5ANGYGFX6A, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:104.36.82.49:4978): protocol: write tcp 159.223.143.39:53376->104.36.82.49:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508743353 (1h15m56.289676732s) [noise=true , address=104.36.82.12:4182 , piece_id=7BUAGBJEH6U35GFWZJZ2ALOZDWJY6IHRTVUSQIH5P3WCZ57GFDWQ, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1KSPtVhJPKYAb3Gj25ihyExNgkDCGo9QTDaFcEfXRfLdDz8nsh, address:104.36.82.12:4182): protocol: write tcp 159.223.143.39:43074->104.36.82.12:4182: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508779785 (1h15m56.289713162s) [noise=true , address=104.36.82.22:4535 , piece_id=TIEDRQ7T5XR5RIY3YZE46JTWEIZJSNNIGTGGBFX5EY3FC2WU5I4A, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 159.223.143.39:50904->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508918006 (1h15m56.289851394s) [noise=true , address=104.36.82.15:4278 , piece_id=PRNJLZPCB67YIVT3HXXTAQTYDCF4B7ITENJ4IHP5CF2GQ5IBTVOA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12qpbyKnJ8qXmos7gdxd46F8QXx98LUwtKD6mryaXPBDD5WFSwu, address:104.36.82.15:4278): protocol: rpc: dial tcp 104.36.82.15:4278: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.508993576 (1h15m56.289926958s) [noise=true , address=104.36.82.4:4835 , piece_id=6ONX3EUIHN3ZV3ZXKLYPMMPPMNKDSGCPCR3I2SQZDJG6ODZIWVAA, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:12WYQwhW2KDjDyweEx5FLYRkGwvW15f6eVHzbdrysiHeuKNtaoN, address:104.36.82.4:4835): protocol: rpc: dial tcp 104.36.82.4:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.509012114 (1h15m56.289945491s) [noise=true , address=104.36.82.52:4091 , piece_id=A36364UTYE5BWAJLV7EBHMKEA7LXHEYSFPPKXBFGGLKQFRGEBL6Q, seg_pos={0 0}, upload=1157]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: write tcp 159.223.143.39:41692->104.36.82.52:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:40.509054458 (1h15m56.289987826s) [seg_pos={0 0}, upload=1157]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 23:07:40.534043629 (1h15m56.314977014s) [upload=1157]: Done uploading segments... Jul 26 23:07:40.534076653 (1h15m56.315010056s) [upload=1157]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.029295627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X................X........................X......X] duration: 30.07114153s FAIL (2025-07-26 23:07:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.071174977s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 334.632677ms FAIL (2025-07-26 23:06:56 UTC) orbiter warning: uplink: too many requests Duration: 334.659199ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.....X.X.X...XX.................X.X..........XX...X...X..XX.] duration: 1.851735979s FAIL (2025-07-26 23:06:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.851778442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [......X.X.X...XX.................X.X..........XX...X...X..XX] duration: 1m0.000637893s FAIL (2025-07-26 23:06:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000660082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 760.79387ms FAIL (2025-07-26 23:06:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 760.936161ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.X] duration: 1m0.051609237s FAIL (2025-07-26 23:06:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== oad=1141]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:05:28.628594097 (1h13m44.409527477s) [seg_pos={0 0}, upload=1141]: Begin upload segment... Jul 26 23:05:28.628694608 (1h13m44.409627996s) [noise=true , address=77.90.31.11:4565 , piece_id=3LZX5FCWTPBPO626BNXBHOLLBXX2DTQ5WBMPE6HBOAZZLNZOUXKA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.629244531 (1h13m44.41017792s) [seg_pos={0 0}, upload=1141]: Done begin upload segment. Jul 26 23:05:28.629283339 (1h13m44.410216725s) [noise=true , address=156.146.43.225:4449 , piece_id=UU3IT2XIESCSKQBDSBWRAZL4TAU4JYVXD3FVHPFCAZ7LI4FM2BQA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.629359950 (1h13m44.410293327s) [noise=true , address=79.127.219.47:4048 , piece_id=MOYSRZRTPDXDIL7OIR2AP72WPD6462GKBQ2NOWU364EW3FW3TK4A, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.631629535 (1h13m44.412562929s) [noise=true , address=45.91.116.11:4151 , piece_id=SAYXV67WSPMUNBXBAYEUBU2KDEVJ6Z7DSWGOTETLC65Q47JWPMOA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.631877702 (1h13m44.412811085s) [noise=true , address=79.127.219.38:4017 , piece_id=ULNTEQB7GSZXRVQ4J2H6VYAQNUFP6UNSOBXL3TKWOHWN5R5QF5ZA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.633078788 (1h13m44.414012177s) [noise=true , address=45.91.116.14:4206 , piece_id=FSRAFGVLLUL6K7GCMQSUTLN2CPDHYZY6SLV45A72CI27VF4IFOPQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.637171961 (1h13m44.418105351s) [noise=true , address=79.127.205.245:4223 , piece_id=I64JI4ALYKNI6AZYSHKEXUS3ZRMEUA52ZOHUZ7LZI6SYP7GFJECQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.637820761 (1h13m44.41875415s) [noise=true , address=79.127.219.51:4210 , piece_id=RP7FUKX3U33FZAMWG72S2N3OV4NWZRP3TBR3CIPE7FNXQ3CEPXVA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.638393389 (1h13m44.419326781s) [noise=true , address=77.90.31.11:4587 , piece_id=4S5GNVLB24GSNOTFW6TZSZIBPLCLFPOVFBBLTV4N36WIXQGQGF7Q, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.639013958 (1h13m44.419947347s) [noise=true , address=45.91.116.5:4310 , piece_id=NOWAW7ZMMBYPZ2ZPRHQ3BWRIMTR7ETC6GAOQJT6GCWLZ6HQREEKA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.643890927 (1h13m44.424824315s) [noise=true , address=77.90.31.17:4774 , piece_id=77GYDBWYEOEEPU53D6OQCWIEOXPJ56O7MH3BLUPM4ABFDOQNU5XQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.644616235 (1h13m44.425549625s) [noise=true , address=79.127.201.214:4321 , piece_id=SFQ2776UVZOKYK3H6A5D3RLHL2VMAWD2S24REJSBIUH4WJFEI4JQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.645154693 (1h13m44.426088079s) [noise=true , address=64.49.13.138:4482 , piece_id=XN4FDKAFB5NZ3P2VUXJVEMT3QRLJMPZW6YBI7C4KYFXC22MPCESQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.645620447 (1h13m44.426553836s) [noise=true , address=156.146.43.225:4434 , piece_id=I26LVHIPQATHFFGEFKRB3DFSFX4SFE6RSXCVOMEIRDEGMOIYOVWA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.646080005 (1h13m44.427013392s) [noise=true , address=45.91.116.14:4222 , piece_id=O3B6MN4QCCMDV7XBYGUB2JW2SPCZ7FTX7SNKJKVWLFX4TXSR7QFQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.646528660 (1h13m44.427462044s) [noise=true , address=45.91.116.9:4277 , piece_id=C4NTD6SJIAZ6SKVUOXZYCU3B6TADZXTBZUIXBTDQLALKRF6MZBFA, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.646938102 (1h13m44.427871491s) [noise=true , address=64.49.13.139:4478 , piece_id=X54V5YMKRWULPYSZTYJ4EGEHZFGVZ5SRQHTTQW56PLSNHGB7HBWQ, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.647785833 (1h13m44.42871922s) [noise=true , address=79.127.205.234:4614 , piece_id=HOACXUX656IWAZXPGSMH6EQQJB7FZ3NK4MJSLW5M62MCKLBGVA5Q, seg_pos={0 0}, upload=1141]: Uploading piece... Jul 26 23:05:28.710076062 (1h13m44.491009442s) [noise=true , address=79.127.219.47:4048 , piece_id=MOYSRZRTPDXDIL7OIR2AP72WPD6462GKBQ2NOWU364EW3FW3TK4A, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:28.739826655 (1h13m44.520760032s) [upload=1141]: Next returned nil segment Jul 26 23:05:28.739849485 (1h13m44.520782852s) [upload=1141]: Waiting for error group managing segments... Jul 26 23:05:28.810592579 (1h13m44.591525978s) [noise=true , address=156.146.43.225:4434 , piece_id=I26LVHIPQATHFFGEFKRB3DFSFX4SFE6RSXCVOMEIRDEGMOIYOVWA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:28.848581660 (1h13m44.629515041s) [noise=true , address=79.127.219.51:4210 , piece_id=RP7FUKX3U33FZAMWG72S2N3OV4NWZRP3TBR3CIPE7FNXQ3CEPXVA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:28.997405787 (1h13m44.778339151s) [noise=true , address=79.127.219.38:4017 , piece_id=ULNTEQB7GSZXRVQ4J2H6VYAQNUFP6UNSOBXL3TKWOHWN5R5QF5ZA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:29.269908065 (1h13m45.050841438s) [noise=true , address=79.127.205.234:4614 , piece_id=HOACXUX656IWAZXPGSMH6EQQJB7FZ3NK4MJSLW5M62MCKLBGVA5Q, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:29.327078430 (1h13m45.108011811s) [noise=true , address=79.127.205.245:4223 , piece_id=I64JI4ALYKNI6AZYSHKEXUS3ZRMEUA52ZOHUZ7LZI6SYP7GFJECQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:29.740795414 (1h13m45.521728784s) [noise=true , address=64.49.13.139:4478 , piece_id=X54V5YMKRWULPYSZTYJ4EGEHZFGVZ5SRQHTTQW56PLSNHGB7HBWQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:30.854867267 (1h13m46.635800653s) [noise=true , address=79.127.201.214:4321 , piece_id=SFQ2776UVZOKYK3H6A5D3RLHL2VMAWD2S24REJSBIUH4WJFEI4JQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:32.001768315 (1h13m47.782701697s) [noise=true , address=156.146.43.225:4449 , piece_id=UU3IT2XIESCSKQBDSBWRAZL4TAU4JYVXD3FVHPFCAZ7LI4FM2BQA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:32.710987582 (1h13m48.491920961s) [noise=true , address=64.49.13.138:4482 , piece_id=XN4FDKAFB5NZ3P2VUXJVEMT3QRLJMPZW6YBI7C4KYFXC22MPCESQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:37.131083709 (1h13m52.912017078s) [noise=true , address=45.91.116.11:4151 , piece_id=SAYXV67WSPMUNBXBAYEUBU2KDEVJ6Z7DSWGOTETLC65Q47JWPMOA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:39.069558528 (1h13m54.850491928s) [noise=true , address=45.91.116.9:4277 , piece_id=C4NTD6SJIAZ6SKVUOXZYCU3B6TADZXTBZUIXBTDQLALKRF6MZBFA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:39.211670250 (1h13m54.992603615s) [noise=true , address=45.91.116.14:4222 , piece_id=O3B6MN4QCCMDV7XBYGUB2JW2SPCZ7FTX7SNKJKVWLFX4TXSR7QFQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:05:40.339113657 (1h13m56.12004703s) [noise=true , address=45.91.116.5:4310 , piece_id=NOWAW7ZMMBYPZ2ZPRHQ3BWRIMTR7ETC6GAOQJT6GCWLZ6HQREEKA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:06:09.158854012 (1h14m24.939787384s) [noise=true , address=77.90.31.17:4774 , piece_id=77GYDBWYEOEEPU53D6OQCWIEOXPJ56O7MH3BLUPM4ABFDOQNU5XQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: <nil> Jul 26 23:06:15.253853134 (1h14m31.034786518s) [noise=true , address=45.91.116.14:4206 , piece_id=FSRAFGVLLUL6K7GCMQSUTLN2CPDHYZY6SLV45A72CI27VF4IFOPQ, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:06:28.443449991 (1h14m44.224383381s) [noise=true , address=77.90.31.11:4587 , piece_id=4S5GNVLB24GSNOTFW6TZSZIBPLCLFPOVFBBLTV4N36WIXQGQGF7Q, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: write tcp 159.223.143.39:41962->77.90.31.11:4587: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:28.443565474 (1h14m44.224498859s) [noise=true , address=77.90.31.11:4565 , piece_id=3LZX5FCWTPBPO626BNXBHOLLBXX2DTQ5WBMPE6HBOAZZLNZOUXKA, seg_pos={0 0}, upload=1141]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: rpc: dial tcp 77.90.31.11:4565: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:28.443604917 (1h14m44.224538297s) [seg_pos={0 0}, upload=1141]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:06:28.492699374 (1h14m44.273632744s) [upload=1141]: Done uploading segments... Jul 26 23:06:28.492731778 (1h14m44.273665147s) [upload=1141]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration:* 1m0.051632624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X................X........................X......] duration: 1.068886634s FAIL (2025-07-26 23:05:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.068921703s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 753.77877ms FAIL (2025-07-26 23:05:44 UTC) orbiter warning: uplink: too many requests Duration: 753.812407ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.......X.X.X...XX.................X.X..........XX...X...X..X] duration: 1m0.000890462s FAIL (2025-07-26 23:05:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000938371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 428.613764ms FAIL (2025-07-26 23:05:29 UTC) orbiter warning: uplink: too many requests Duration: 428.641952ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.] duration: 17.874108114s FAIL (2025-07-26 23:05:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 6 23:04:10.417101156 (1h12m26.198034544s) [noise=true , address=64.49.13.170:4721 , piece_id=CGTS52RD75J26U7E6565KNMIYREQGALBBUT67HSPNOCIKKECP27Q, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.417726595 (1h12m26.198659981s) [noise=true , address=79.127.205.246:4286 , piece_id=WOLUWABJMUMNAESMIMWMI4YN3RRIZURUBIWZ6U5RTWBXX5XCCMHA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.417914592 (1h12m26.198847979s) [noise=true , address=45.91.116.9:4267 , piece_id=6KF7SNXUUMPX4Q2ZNAEZISZWUSZYGS4OJWELEWWUXLT4CXTVHSRA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.418823898 (1h12m26.199757285s) [noise=true , address=77.90.31.19:4849 , piece_id=ZLB3QJI4UKMPD63VIQGUCKVV2FPVOVUBB7NNUTLOE6K3IFFSCAFA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.418996906 (1h12m26.199930268s) [noise=true , address=79.127.205.234:4606 , piece_id=QEIPNOBWE5M54ZYSFK5BRKPWQOJX4P2MAQAPZD4VC43G6EFTBQLQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.419970568 (1h12m26.200903957s) [noise=true , address=79.127.219.38:4015 , piece_id=UP2XW2FKWMARLG7ENH4EO3USUW3UXHNUYMPYK6UKH255D2FIKKIQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.420440076 (1h12m26.201373465s) [noise=true , address=79.127.205.245:4250 , piece_id=A7JVM6U3RNXMIKH3WVVFDMVDZMKH52M5FMSU3VYAMH37RQXW7TKQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.420989928 (1h12m26.201923315s) [noise=true , address=79.127.201.214:4321 , piece_id=NX5MXQPVNCW76I6ISOOBF27QA7OR2KECOGIFL4VLH6RNP42OBQRQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.421356482 (1h12m26.202289873s) [noise=true , address=45.91.116.9:4277 , piece_id=BNCMSJSE2F2T32VVMDKS2U3XELOZ5Q6X73PPW473K24ZTUTHCOYQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.421716261 (1h12m26.202649651s) [noise=true , address=79.127.205.233:4633 , piece_id=HMQEFEO2R2FVUH5JTYAON5LZ742NKKDQTRAUS6JTTYPLQ46VN5PA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.422228566 (1h12m26.203161955s) [noise=true , address=77.90.31.11:4579 , piece_id=UCLB37RQG6HXD22JG7P4FOTZ4GZLLO3BXX4BHNMOZFRDQATSPGNA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.422867505 (1h12m26.203800898s) [noise=true , address=79.127.219.51:4212 , piece_id=NHGV7UAQNB5O5I4YLZJZVERMDOS4MA7HDMW4VKC6AKVH3VVVZQZQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.423282554 (1h12m26.204215936s) [noise=true , address=64.49.13.155:4923 , piece_id=BWE6UJP4MJZ5K6W4Q4RHJZLWJ6UHHLXEXGXHDHMYLCEL76MTG2QA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.423768254 (1h12m26.204701641s) [noise=true , address=79.127.220.101:4048 , piece_id=6IGLNB4WEHM54FXOAW6XPGDVZYVPUXW4ASRD42ODAQ7EESODFYIQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.424451766 (1h12m26.20538513s) [noise=true , address=79.127.201.215:4354 , piece_id=VDSREHTXXWNCHS4NHZAY56XOUMWZ3OSEJ2PT6ISISLEPZMOWRSTQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.431524809 (1h12m26.212458194s) [noise=true , address=77.90.31.15:4717 , piece_id=N33SAR7V77HGJDDVFY4C5BPRLN4ZFDLZFSGGMOP3KSXLLUDROQCQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.433610399 (1h12m26.214543773s) [noise=true , address=77.90.31.16:4739 , piece_id=2KRWF3O2EM3YWCYHRETO5RY3XDXP6EP7JGDDUG3OVERX3TWO35RQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.435563240 (1h12m26.216496606s) [noise=true , address=64.49.13.140:4514 , piece_id=UWICCTLD5HLCB2CKXXG53T3OIKQ5LON2BNT4FVA4TC74OIZ2X5IA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.463289025 (1h12m26.244222393s) [noise=true , address=79.127.219.51:4212 , piece_id=NHGV7UAQNB5O5I4YLZJZVERMDOS4MA7HDMW4VKC6AKVH3VVVZQZQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:10.500486501 (1h12m26.28141989s) [upload=1121]: Next returned nil segment Jul 26 23:04:10.500511419 (1h12m26.28144482s) [upload=1121]: Waiting for error group managing segments... Jul 26 23:04:10.513835566 (1h12m26.294768931s) [noise=true , address=79.127.219.38:4015 , piece_id=UP2XW2FKWMARLG7ENH4EO3USUW3UXHNUYMPYK6UKH255D2FIKKIQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:10.585488550 (1h12m26.366421933s) [noise=true , address=79.127.201.215:4354 , piece_id=VDSREHTXXWNCHS4NHZAY56XOUMWZ3OSEJ2PT6ISISLEPZMOWRSTQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:13Jtv8kL5R9oVro2D4ioqsznoJBXyNgJLdj1FcrGN3WWH45rkP, address:79.127.201.215:4354): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:10.593364447 (1h12m26.374297834s) [noise=true , address=79.127.220.101:4048 , piece_id=6IGLNB4WEHM54FXOAW6XPGDVZYVPUXW4ASRD42ODAQ7EESODFYIQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:11.029127822 (1h12m26.810061191s) [noise=true , address=79.127.205.246:4286 , piece_id=WOLUWABJMUMNAESMIMWMI4YN3RRIZURUBIWZ6U5RTWBXX5XCCMHA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.030089513 (1h12m26.81102289s) [noise=true , address=79.127.205.245:4250 , piece_id=A7JVM6U3RNXMIKH3WVVFDMVDZMKH52M5FMSU3VYAMH37RQXW7TKQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.039837172 (1h12m26.820770535s) [noise=true , address=79.127.205.233:4633 , piece_id=HMQEFEO2R2FVUH5JTYAON5LZ742NKKDQTRAUS6JTTYPLQ46VN5PA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.049753426 (1h12m26.830686806s) [noise=true , address=79.127.205.234:4606 , piece_id=QEIPNOBWE5M54ZYSFK5BRKPWQOJX4P2MAQAPZD4VC43G6EFTBQLQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:12.300355942 (1h12m28.081289319s) [noise=true , address=79.127.201.214:4321 , piece_id=NX5MXQPVNCW76I6ISOOBF27QA7OR2KECOGIFL4VLH6RNP42OBQRQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:12.337009638 (1h12m28.117943032s) [noise=true , address=64.49.13.140:4514 , piece_id=UWICCTLD5HLCB2CKXXG53T3OIKQ5LON2BNT4FVA4TC74OIZ2X5IA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:13.175933117 (1h12m28.956866501s) [noise=true , address=64.49.13.170:4721 , piece_id=CGTS52RD75J26U7E6565KNMIYREQGALBBUT67HSPNOCIKKECP27Q, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:13.280233882 (1h12m29.061167268s) [noise=true , address=64.49.13.155:4923 , piece_id=BWE6UJP4MJZ5K6W4Q4RHJZLWJ6UHHLXEXGXHDHMYLCEL76MTG2QA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:14.871557683 (1h12m30.652491078s) [noise=true , address=45.91.116.9:4267 , piece_id=6KF7SNXUUMPX4Q2ZNAEZISZWUSZYGS4OJWELEWWUXLT4CXTVHSRA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:18.580092593 (1h12m34.361025956s) [noise=true , address=45.91.116.9:4277 , piece_id=BNCMSJSE2F2T32VVMDKS2U3XELOZ5Q6X73PPW473K24ZTUTHCOYQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:05:10.332106674 (1h13m26.113040057s) [noise=true , address=77.90.31.19:4849 , piece_id=ZLB3QJI4UKMPD63VIQGUCKVV2FPVOVUBB7NNUTLOE6K3IFFSCAFA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:05:10.332198466 (1h13m26.113131843s) [noise=true , address=77.90.31.16:4739 , piece_id=2KRWF3O2EM3YWCYHRETO5RY3XDXP6EP7JGDDUG3OVERX3TWO35RQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: write tcp 159.223.143.39:49606->77.90.31.16:4739: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:05:10.332304942 (1h13m26.113238319s) [noise=true , address=77.90.31.11:4579 , piece_id=UCLB37RQG6HXD22JG7P4FOTZ4GZLLO3BXX4BHNMOZFRDQATSPGNA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:05:10.332387108 (1h13m26.113320485s) [noise=true , address=77.90.31.15:4717 , piece_id=N33SAR7V77HGJDDVFY4C5BPRLN4ZFDLZFSGGMOP3KSXLLUDROQCQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:05:10.332414088 (1h13m26.113347476s) [seg_pos={0 0}, upload=1121]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:05:10.417384543 (1h13m26.19831792s) [upload=1121]: Done uploading segments... Jul 26 23:05:10.417409881 (1h13m26.19834325s) [upload=1121]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 17.874133346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X................X........................X.....] duration: 672.995755ms FAIL (2025-07-26 23:05:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 673.087823ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........X..................................................] duration: 768.238468ms FAIL (2025-07-26 23:05:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 768.271042ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X] duration: 1m0.087427744s FAIL (2025-07-26 23:05:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 6 23:04:10.417101156 (1h12m26.198034544s) [noise=true , address=64.49.13.170:4721 , piece_id=CGTS52RD75J26U7E6565KNMIYREQGALBBUT67HSPNOCIKKECP27Q, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.417726595 (1h12m26.198659981s) [noise=true , address=79.127.205.246:4286 , piece_id=WOLUWABJMUMNAESMIMWMI4YN3RRIZURUBIWZ6U5RTWBXX5XCCMHA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.417914592 (1h12m26.198847979s) [noise=true , address=45.91.116.9:4267 , piece_id=6KF7SNXUUMPX4Q2ZNAEZISZWUSZYGS4OJWELEWWUXLT4CXTVHSRA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.418823898 (1h12m26.199757285s) [noise=true , address=77.90.31.19:4849 , piece_id=ZLB3QJI4UKMPD63VIQGUCKVV2FPVOVUBB7NNUTLOE6K3IFFSCAFA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.418996906 (1h12m26.199930268s) [noise=true , address=79.127.205.234:4606 , piece_id=QEIPNOBWE5M54ZYSFK5BRKPWQOJX4P2MAQAPZD4VC43G6EFTBQLQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.419970568 (1h12m26.200903957s) [noise=true , address=79.127.219.38:4015 , piece_id=UP2XW2FKWMARLG7ENH4EO3USUW3UXHNUYMPYK6UKH255D2FIKKIQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.420440076 (1h12m26.201373465s) [noise=true , address=79.127.205.245:4250 , piece_id=A7JVM6U3RNXMIKH3WVVFDMVDZMKH52M5FMSU3VYAMH37RQXW7TKQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.420989928 (1h12m26.201923315s) [noise=true , address=79.127.201.214:4321 , piece_id=NX5MXQPVNCW76I6ISOOBF27QA7OR2KECOGIFL4VLH6RNP42OBQRQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.421356482 (1h12m26.202289873s) [noise=true , address=45.91.116.9:4277 , piece_id=BNCMSJSE2F2T32VVMDKS2U3XELOZ5Q6X73PPW473K24ZTUTHCOYQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.421716261 (1h12m26.202649651s) [noise=true , address=79.127.205.233:4633 , piece_id=HMQEFEO2R2FVUH5JTYAON5LZ742NKKDQTRAUS6JTTYPLQ46VN5PA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.422228566 (1h12m26.203161955s) [noise=true , address=77.90.31.11:4579 , piece_id=UCLB37RQG6HXD22JG7P4FOTZ4GZLLO3BXX4BHNMOZFRDQATSPGNA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.422867505 (1h12m26.203800898s) [noise=true , address=79.127.219.51:4212 , piece_id=NHGV7UAQNB5O5I4YLZJZVERMDOS4MA7HDMW4VKC6AKVH3VVVZQZQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.423282554 (1h12m26.204215936s) [noise=true , address=64.49.13.155:4923 , piece_id=BWE6UJP4MJZ5K6W4Q4RHJZLWJ6UHHLXEXGXHDHMYLCEL76MTG2QA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.423768254 (1h12m26.204701641s) [noise=true , address=79.127.220.101:4048 , piece_id=6IGLNB4WEHM54FXOAW6XPGDVZYVPUXW4ASRD42ODAQ7EESODFYIQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.424451766 (1h12m26.20538513s) [noise=true , address=79.127.201.215:4354 , piece_id=VDSREHTXXWNCHS4NHZAY56XOUMWZ3OSEJ2PT6ISISLEPZMOWRSTQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.431524809 (1h12m26.212458194s) [noise=true , address=77.90.31.15:4717 , piece_id=N33SAR7V77HGJDDVFY4C5BPRLN4ZFDLZFSGGMOP3KSXLLUDROQCQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.433610399 (1h12m26.214543773s) [noise=true , address=77.90.31.16:4739 , piece_id=2KRWF3O2EM3YWCYHRETO5RY3XDXP6EP7JGDDUG3OVERX3TWO35RQ, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.435563240 (1h12m26.216496606s) [noise=true , address=64.49.13.140:4514 , piece_id=UWICCTLD5HLCB2CKXXG53T3OIKQ5LON2BNT4FVA4TC74OIZ2X5IA, seg_pos={0 0}, upload=1121]: Uploading piece... Jul 26 23:04:10.463289025 (1h12m26.244222393s) [noise=true , address=79.127.219.51:4212 , piece_id=NHGV7UAQNB5O5I4YLZJZVERMDOS4MA7HDMW4VKC6AKVH3VVVZQZQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:10.500486501 (1h12m26.28141989s) [upload=1121]: Next returned nil segment Jul 26 23:04:10.500511419 (1h12m26.28144482s) [upload=1121]: Waiting for error group managing segments... Jul 26 23:04:10.513835566 (1h12m26.294768931s) [noise=true , address=79.127.219.38:4015 , piece_id=UP2XW2FKWMARLG7ENH4EO3USUW3UXHNUYMPYK6UKH255D2FIKKIQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:10.585488550 (1h12m26.366421933s) [noise=true , address=79.127.201.215:4354 , piece_id=VDSREHTXXWNCHS4NHZAY56XOUMWZ3OSEJ2PT6ISISLEPZMOWRSTQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:13Jtv8kL5R9oVro2D4ioqsznoJBXyNgJLdj1FcrGN3WWH45rkP, address:79.127.201.215:4354): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:10.593364447 (1h12m26.374297834s) [noise=true , address=79.127.220.101:4048 , piece_id=6IGLNB4WEHM54FXOAW6XPGDVZYVPUXW4ASRD42ODAQ7EESODFYIQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:11.029127822 (1h12m26.810061191s) [noise=true , address=79.127.205.246:4286 , piece_id=WOLUWABJMUMNAESMIMWMI4YN3RRIZURUBIWZ6U5RTWBXX5XCCMHA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.030089513 (1h12m26.81102289s) [noise=true , address=79.127.205.245:4250 , piece_id=A7JVM6U3RNXMIKH3WVVFDMVDZMKH52M5FMSU3VYAMH37RQXW7TKQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.039837172 (1h12m26.820770535s) [noise=true , address=79.127.205.233:4633 , piece_id=HMQEFEO2R2FVUH5JTYAON5LZ742NKKDQTRAUS6JTTYPLQ46VN5PA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:11.049753426 (1h12m26.830686806s) [noise=true , address=79.127.205.234:4606 , piece_id=QEIPNOBWE5M54ZYSFK5BRKPWQOJX4P2MAQAPZD4VC43G6EFTBQLQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:12.300355942 (1h12m28.081289319s) [noise=true , address=79.127.201.214:4321 , piece_id=NX5MXQPVNCW76I6ISOOBF27QA7OR2KECOGIFL4VLH6RNP42OBQRQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:12.337009638 (1h12m28.117943032s) [noise=true , address=64.49.13.140:4514 , piece_id=UWICCTLD5HLCB2CKXXG53T3OIKQ5LON2BNT4FVA4TC74OIZ2X5IA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:13.175933117 (1h12m28.956866501s) [noise=true , address=64.49.13.170:4721 , piece_id=CGTS52RD75J26U7E6565KNMIYREQGALBBUT67HSPNOCIKKECP27Q, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:13.280233882 (1h12m29.061167268s) [noise=true , address=64.49.13.155:4923 , piece_id=BWE6UJP4MJZ5K6W4Q4RHJZLWJ6UHHLXEXGXHDHMYLCEL76MTG2QA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:04:14.871557683 (1h12m30.652491078s) [noise=true , address=45.91.116.9:4267 , piece_id=6KF7SNXUUMPX4Q2ZNAEZISZWUSZYGS4OJWELEWWUXLT4CXTVHSRA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:18.580092593 (1h12m34.361025956s) [noise=true , address=45.91.116.9:4277 , piece_id=BNCMSJSE2F2T32VVMDKS2U3XELOZ5Q6X73PPW473K24ZTUTHCOYQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: <nil> Jul 26 23:05:10.332106674 (1h13m26.113040057s) [noise=true , address=77.90.31.19:4849 , piece_id=ZLB3QJI4UKMPD63VIQGUCKVV2FPVOVUBB7NNUTLOE6K3IFFSCAFA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:05:10.332198466 (1h13m26.113131843s) [noise=true , address=77.90.31.16:4739 , piece_id=2KRWF3O2EM3YWCYHRETO5RY3XDXP6EP7JGDDUG3OVERX3TWO35RQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: write tcp 159.223.143.39:49606->77.90.31.16:4739: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:05:10.332304942 (1h13m26.113238319s) [noise=true , address=77.90.31.11:4579 , piece_id=UCLB37RQG6HXD22JG7P4FOTZ4GZLLO3BXX4BHNMOZFRDQATSPGNA, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:05:10.332387108 (1h13m26.113320485s) [noise=true , address=77.90.31.15:4717 , piece_id=N33SAR7V77HGJDDVFY4C5BPRLN4ZFDLZFSGGMOP3KSXLLUDROQCQ, seg_pos={0 0}, upload=1121]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:05:10.332414088 (1h13m26.113347476s) [seg_pos={0 0}, upload=1121]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:05:10.417384543 (1h13m26.19831792s) [upload=1121]: Done uploading segments... Jul 26 23:05:10.417409881 (1h13m26.19834325s) [upload=1121]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.087458277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X................X........................X....] duration: 636.182191ms FAIL (2025-07-26 23:04:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 636.252781ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X........X.................................................] duration: 462.208902ms FAIL (2025-07-26 23:04:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.238368ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 197.952963ms FAIL (2025-07-26 23:04:16 UTC) orbiter warning: uplink: too many requests Duration: 197.986264ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............X................X........................X...] duration: 670.712401ms FAIL (2025-07-26 23:03:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 670.744791ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.] duration: 11.38683419s FAIL (2025-07-26 23:03:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 639916s) [upload=1083]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Jul 26 23:02:38.009431979 (1h10m53.790365367s) [upload=1100]: Starting upload Jul 26 23:02:38.009529742 (1h10m53.79046312s) [upload=1100]: Done starting upload Jul 26 23:02:38.009693140 (1h10m53.790626551s) [upload=1100]: Uploading segments... Jul 26 23:02:38.009988782 (1h10m53.790922148s) [upload=1100]: Got next segment. Inline: false Jul 26 23:02:38.010070015 (1h10m53.7910034s) [upload=1100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:02:38.010077679 (1h10m53.791011064s) [upload=1100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:02:38.044231887 (1h10m53.825165274s) [seg_pos={0 0}, upload=1100]: Begin upload segment... Jul 26 23:02:38.044288604 (1h10m53.825221986s) [seg_pos={0 0}, upload=1100]: Done begin upload segment. Jul 26 23:02:38.044317043 (1h10m53.82525043s) [noise=true , address=64.49.13.139:4478 , piece_id=3GN3HISCZ4EEIID2GIUWAWGWILQKIQEDN25T5OSFTFCE3RLWOBKA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044360663 (1h10m53.825294057s) [noise=true , address=45.91.116.11:4121 , piece_id=P76COMVRFHS4DLD45V5HWKZRLJDKYPL3M2IAPKDXUT4X2F6KV3CQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044838801 (1h10m53.825772189s) [noise=true , address=77.90.31.19:4859 , piece_id=OIKITA4V76KN3VP7KPWZYHJTRA66RFFV5HFRWGEW46YLWIKZTUIA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044953426 (1h10m53.82588681s) [noise=true , address=79.127.219.47:4061 , piece_id=FR5RPPWQ74GS7GLFKSQMAQ4HB2UTI6ZNKKGEGYNITPEE35KUTALA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045307782 (1h10m53.826241184s) [noise=true , address=79.127.205.245:4243 , piece_id=KIBMUQOZG6EJFBWRKC5ZAIUJCKZXDH5GA4OJIK6M2JVXX76XT5SA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045577917 (1h10m53.826511306s) [noise=true , address=79.127.205.234:4615 , piece_id=6DYCMDDTBMYEWJ5DMSKR5OBKD4SU7ZSKE3O4T4HZ6QRUTFN3GVJA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045795589 (1h10m53.826728975s) [noise=true , address=45.91.116.9:4277 , piece_id=QWAN66EE7R54K2TZYJRFBBP3WHH3CVYG4UTJ2BCDSVKXRFSXBM2A, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046160990 (1h10m53.827094381s) [noise=true , address=79.127.205.235:4685 , piece_id=D2WL6BO4WMWX2VRVQ2V6TQIQPRONVJCWXZ4SGAM6DQ4CU3R3QELA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046168935 (1h10m53.827102298s) [noise=true , address=79.127.220.101:4048 , piece_id=F2NCRBZL4AVI6QAS2OQ6DBBSBEFMBT4TARCTZWJOKINI65VTMZHQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046544662 (1h10m53.827478048s) [noise=true , address=77.90.31.15:4727 , piece_id=6E7RVTU3GRXXXIOJWRIXW36JASGTOOYJ7JTG5VUUGGNJTJQKIRHA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046638748 (1h10m53.82757213s) [noise=true , address=79.127.219.48:4090 , piece_id=CZCMAKDM3UMRVCH226TCKAMN5PKHH3KRWQPAENHTVKKU2AANJGAA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046971631 (1h10m53.827905015s) [noise=true , address=45.91.116.14:4206 , piece_id=GS53BPTRXHDKND5G5CI7QVK2LMOTA2AGSBI3RQSDKLMY5UYWWYFA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.047469479 (1h10m53.828402863s) [noise=true , address=79.127.219.38:4004 , piece_id=I4N3UNU4PMU6GRS3ZVO6FYYYXSLLXSI45GBXJN3KORIAIDLKNA2A, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.047886394 (1h10m53.828819784s) [noise=true , address=77.90.31.11:4571 , piece_id=QVQ45R3X7IMH22OC3MTV3MJC6YQSCVWNEEZB3266TPDSB2NDZAKQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048312600 (1h10m53.829245988s) [noise=true , address=79.127.205.245:4224 , piece_id=3ST6YVMWDM2D45WIGTCONJH4MALS3STDK5BSHVPSCOFPKA65DHZQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048741550 (1h10m53.829674942s) [noise=true , address=77.90.31.12:4608 , piece_id=5GNVVBTWM75DK7J75ARYEA7Y6HBCS5UJJJK6E3J6ZFRVAYBEFY3Q, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048867751 (1h10m53.829801138s) [noise=true , address=79.127.201.215:4358 , piece_id=RRICN3R6RRF2MIBZM2ZXIXI6NG2QKRAAMCTZ5ZUQOQBRHHHWANDQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.049276904 (1h10m53.830210291s) [noise=true , address=79.127.219.51:4212 , piece_id=2V5P22DCZLVE5VSEKPISV2RZBFRBCUSFD3CG63IHKUZLIT56LORA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.113183749 (1h10m53.894117126s) [noise=true , address=79.127.219.51:4212 , piece_id=2V5P22DCZLVE5VSEKPISV2RZBFRBCUSFD3CG63IHKUZLIT56LORA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:38.135417853 (1h10m53.916351218s) [upload=1100]: Next returned nil segment Jul 26 23:02:38.135441335 (1h10m53.916374722s) [upload=1100]: Waiting for error group managing segments... Jul 26 23:02:38.296608477 (1h10m54.077541863s) [noise=true , address=79.127.219.38:4004 , piece_id=I4N3UNU4PMU6GRS3ZVO6FYYYXSLLXSI45GBXJN3KORIAIDLKNA2A, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.553667152 (1h10m54.334600522s) [noise=true , address=77.90.31.12:4608 , piece_id=5GNVVBTWM75DK7J75ARYEA7Y6HBCS5UJJJK6E3J6ZFRVAYBEFY3Q, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.631703695 (1h10m54.412637075s) [noise=true , address=79.127.205.235:4685 , piece_id=D2WL6BO4WMWX2VRVQ2V6TQIQPRONVJCWXZ4SGAM6DQ4CU3R3QELA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.646996395 (1h10m54.427929767s) [noise=true , address=79.127.205.245:4243 , piece_id=KIBMUQOZG6EJFBWRKC5ZAIUJCKZXDH5GA4OJIK6M2JVXX76XT5SA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.660623879 (1h10m54.441557271s) [noise=true , address=79.127.205.245:4224 , piece_id=3ST6YVMWDM2D45WIGTCONJH4MALS3STDK5BSHVPSCOFPKA65DHZQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.661444603 (1h10m54.442377975s) [noise=true , address=79.127.205.234:4615 , piece_id=6DYCMDDTBMYEWJ5DMSKR5OBKD4SU7ZSKE3O4T4HZ6QRUTFN3GVJA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.480167315 (1h10m56.261100702s) [noise=true , address=79.127.220.101:4048 , piece_id=F2NCRBZL4AVI6QAS2OQ6DBBSBEFMBT4TARCTZWJOKINI65VTMZHQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.846007988 (1h10m56.626941368s) [noise=true , address=79.127.219.48:4090 , piece_id=CZCMAKDM3UMRVCH226TCKAMN5PKHH3KRWQPAENHTVKKU2AANJGAA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.905994229 (1h10m56.686927607s) [noise=true , address=79.127.219.47:4061 , piece_id=FR5RPPWQ74GS7GLFKSQMAQ4HB2UTI6ZNKKGEGYNITPEE35KUTALA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:44.563162762 (1h11m0.344096151s) [noise=true , address=45.91.116.14:4206 , piece_id=GS53BPTRXHDKND5G5CI7QVK2LMOTA2AGSBI3RQSDKLMY5UYWWYFA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:45.387863794 (1h11m1.168797163s) [noise=true , address=45.91.116.9:4277 , piece_id=QWAN66EE7R54K2TZYJRFBBP3WHH3CVYG4UTJ2BCDSVKXRFSXBM2A, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:46.641102649 (1h11m2.422036011s) [noise=true , address=64.49.13.139:4478 , piece_id=3GN3HISCZ4EEIID2GIUWAWGWILQKIQEDN25T5OSFTFCE3RLWOBKA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:56.398068001 (1h11m12.179001368s) [noise=true , address=77.90.31.19:4859 , piece_id=OIKITA4V76KN3VP7KPWZYHJTRA66RFFV5HFRWGEW46YLWIKZTUIA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:03:12.478927638 (1h11m28.259861015s) [noise=true , address=45.91.116.11:4121 , piece_id=P76COMVRFHS4DLD45V5HWKZRLJDKYPL3M2IAPKDXUT4X2F6KV3CQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 23:03:22.198648346 (1h11m37.979581727s) [noise=true , address=79.127.201.215:4358 , piece_id=RRICN3R6RRF2MIBZM2ZXIXI6NG2QKRAAMCTZ5ZUQOQBRHHHWANDQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:03:37.964568820 (1h11m53.7455022s) [noise=true , address=77.90.31.11:4571 , piece_id=QVQ45R3X7IMH22OC3MTV3MJC6YQSCVWNEEZB3266TPDSB2NDZAKQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 159.223.143.39:42210->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:37.965835006 (1h11m53.746768388s) [noise=true , address=77.90.31.15:4727 , piece_id=6E7RVTU3GRXXXIOJWRIXW36JASGTOOYJ7JTG5VUUGGNJTJQKIRHA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 159.223.143.39:36482->77.90.31.15:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:37.967314376 (1h11m53.74824777s) [seg_pos={0 0}, upload=1100]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:03:37.994728808 (1h11m53.775662191s) [upload=1100]: Done uploading segments... Jul 26 23:03:37.994772690 (1h11m53.775706073s) [upload=1100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.386861202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [........X.X.X...XX.................X.X..........XX...X...X..] duration: 13.316063426s FAIL (2025-07-26 23:03:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.316242504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX] duration: 1m0.033052521s FAIL (2025-07-26 23:03:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 639916s) [upload=1083]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Jul 26 23:02:38.009431979 (1h10m53.790365367s) [upload=1100]: Starting upload Jul 26 23:02:38.009529742 (1h10m53.79046312s) [upload=1100]: Done starting upload Jul 26 23:02:38.009693140 (1h10m53.790626551s) [upload=1100]: Uploading segments... Jul 26 23:02:38.009988782 (1h10m53.790922148s) [upload=1100]: Got next segment. Inline: false Jul 26 23:02:38.010070015 (1h10m53.7910034s) [upload=1100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:02:38.010077679 (1h10m53.791011064s) [upload=1100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:02:38.044231887 (1h10m53.825165274s) [seg_pos={0 0}, upload=1100]: Begin upload segment... Jul 26 23:02:38.044288604 (1h10m53.825221986s) [seg_pos={0 0}, upload=1100]: Done begin upload segment. Jul 26 23:02:38.044317043 (1h10m53.82525043s) [noise=true , address=64.49.13.139:4478 , piece_id=3GN3HISCZ4EEIID2GIUWAWGWILQKIQEDN25T5OSFTFCE3RLWOBKA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044360663 (1h10m53.825294057s) [noise=true , address=45.91.116.11:4121 , piece_id=P76COMVRFHS4DLD45V5HWKZRLJDKYPL3M2IAPKDXUT4X2F6KV3CQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044838801 (1h10m53.825772189s) [noise=true , address=77.90.31.19:4859 , piece_id=OIKITA4V76KN3VP7KPWZYHJTRA66RFFV5HFRWGEW46YLWIKZTUIA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.044953426 (1h10m53.82588681s) [noise=true , address=79.127.219.47:4061 , piece_id=FR5RPPWQ74GS7GLFKSQMAQ4HB2UTI6ZNKKGEGYNITPEE35KUTALA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045307782 (1h10m53.826241184s) [noise=true , address=79.127.205.245:4243 , piece_id=KIBMUQOZG6EJFBWRKC5ZAIUJCKZXDH5GA4OJIK6M2JVXX76XT5SA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045577917 (1h10m53.826511306s) [noise=true , address=79.127.205.234:4615 , piece_id=6DYCMDDTBMYEWJ5DMSKR5OBKD4SU7ZSKE3O4T4HZ6QRUTFN3GVJA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.045795589 (1h10m53.826728975s) [noise=true , address=45.91.116.9:4277 , piece_id=QWAN66EE7R54K2TZYJRFBBP3WHH3CVYG4UTJ2BCDSVKXRFSXBM2A, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046160990 (1h10m53.827094381s) [noise=true , address=79.127.205.235:4685 , piece_id=D2WL6BO4WMWX2VRVQ2V6TQIQPRONVJCWXZ4SGAM6DQ4CU3R3QELA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046168935 (1h10m53.827102298s) [noise=true , address=79.127.220.101:4048 , piece_id=F2NCRBZL4AVI6QAS2OQ6DBBSBEFMBT4TARCTZWJOKINI65VTMZHQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046544662 (1h10m53.827478048s) [noise=true , address=77.90.31.15:4727 , piece_id=6E7RVTU3GRXXXIOJWRIXW36JASGTOOYJ7JTG5VUUGGNJTJQKIRHA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046638748 (1h10m53.82757213s) [noise=true , address=79.127.219.48:4090 , piece_id=CZCMAKDM3UMRVCH226TCKAMN5PKHH3KRWQPAENHTVKKU2AANJGAA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.046971631 (1h10m53.827905015s) [noise=true , address=45.91.116.14:4206 , piece_id=GS53BPTRXHDKND5G5CI7QVK2LMOTA2AGSBI3RQSDKLMY5UYWWYFA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.047469479 (1h10m53.828402863s) [noise=true , address=79.127.219.38:4004 , piece_id=I4N3UNU4PMU6GRS3ZVO6FYYYXSLLXSI45GBXJN3KORIAIDLKNA2A, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.047886394 (1h10m53.828819784s) [noise=true , address=77.90.31.11:4571 , piece_id=QVQ45R3X7IMH22OC3MTV3MJC6YQSCVWNEEZB3266TPDSB2NDZAKQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048312600 (1h10m53.829245988s) [noise=true , address=79.127.205.245:4224 , piece_id=3ST6YVMWDM2D45WIGTCONJH4MALS3STDK5BSHVPSCOFPKA65DHZQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048741550 (1h10m53.829674942s) [noise=true , address=77.90.31.12:4608 , piece_id=5GNVVBTWM75DK7J75ARYEA7Y6HBCS5UJJJK6E3J6ZFRVAYBEFY3Q, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.048867751 (1h10m53.829801138s) [noise=true , address=79.127.201.215:4358 , piece_id=RRICN3R6RRF2MIBZM2ZXIXI6NG2QKRAAMCTZ5ZUQOQBRHHHWANDQ, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.049276904 (1h10m53.830210291s) [noise=true , address=79.127.219.51:4212 , piece_id=2V5P22DCZLVE5VSEKPISV2RZBFRBCUSFD3CG63IHKUZLIT56LORA, seg_pos={0 0}, upload=1100]: Uploading piece... Jul 26 23:02:38.113183749 (1h10m53.894117126s) [noise=true , address=79.127.219.51:4212 , piece_id=2V5P22DCZLVE5VSEKPISV2RZBFRBCUSFD3CG63IHKUZLIT56LORA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:38.135417853 (1h10m53.916351218s) [upload=1100]: Next returned nil segment Jul 26 23:02:38.135441335 (1h10m53.916374722s) [upload=1100]: Waiting for error group managing segments... Jul 26 23:02:38.296608477 (1h10m54.077541863s) [noise=true , address=79.127.219.38:4004 , piece_id=I4N3UNU4PMU6GRS3ZVO6FYYYXSLLXSI45GBXJN3KORIAIDLKNA2A, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.553667152 (1h10m54.334600522s) [noise=true , address=77.90.31.12:4608 , piece_id=5GNVVBTWM75DK7J75ARYEA7Y6HBCS5UJJJK6E3J6ZFRVAYBEFY3Q, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.631703695 (1h10m54.412637075s) [noise=true , address=79.127.205.235:4685 , piece_id=D2WL6BO4WMWX2VRVQ2V6TQIQPRONVJCWXZ4SGAM6DQ4CU3R3QELA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.646996395 (1h10m54.427929767s) [noise=true , address=79.127.205.245:4243 , piece_id=KIBMUQOZG6EJFBWRKC5ZAIUJCKZXDH5GA4OJIK6M2JVXX76XT5SA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.660623879 (1h10m54.441557271s) [noise=true , address=79.127.205.245:4224 , piece_id=3ST6YVMWDM2D45WIGTCONJH4MALS3STDK5BSHVPSCOFPKA65DHZQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:38.661444603 (1h10m54.442377975s) [noise=true , address=79.127.205.234:4615 , piece_id=6DYCMDDTBMYEWJ5DMSKR5OBKD4SU7ZSKE3O4T4HZ6QRUTFN3GVJA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.480167315 (1h10m56.261100702s) [noise=true , address=79.127.220.101:4048 , piece_id=F2NCRBZL4AVI6QAS2OQ6DBBSBEFMBT4TARCTZWJOKINI65VTMZHQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.846007988 (1h10m56.626941368s) [noise=true , address=79.127.219.48:4090 , piece_id=CZCMAKDM3UMRVCH226TCKAMN5PKHH3KRWQPAENHTVKKU2AANJGAA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:40.905994229 (1h10m56.686927607s) [noise=true , address=79.127.219.47:4061 , piece_id=FR5RPPWQ74GS7GLFKSQMAQ4HB2UTI6ZNKKGEGYNITPEE35KUTALA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:44.563162762 (1h11m0.344096151s) [noise=true , address=45.91.116.14:4206 , piece_id=GS53BPTRXHDKND5G5CI7QVK2LMOTA2AGSBI3RQSDKLMY5UYWWYFA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:45.387863794 (1h11m1.168797163s) [noise=true , address=45.91.116.9:4277 , piece_id=QWAN66EE7R54K2TZYJRFBBP3WHH3CVYG4UTJ2BCDSVKXRFSXBM2A, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:46.641102649 (1h11m2.422036011s) [noise=true , address=64.49.13.139:4478 , piece_id=3GN3HISCZ4EEIID2GIUWAWGWILQKIQEDN25T5OSFTFCE3RLWOBKA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:02:56.398068001 (1h11m12.179001368s) [noise=true , address=77.90.31.19:4859 , piece_id=OIKITA4V76KN3VP7KPWZYHJTRA66RFFV5HFRWGEW46YLWIKZTUIA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:03:12.478927638 (1h11m28.259861015s) [noise=true , address=45.91.116.11:4121 , piece_id=P76COMVRFHS4DLD45V5HWKZRLJDKYPL3M2IAPKDXUT4X2F6KV3CQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 23:03:22.198648346 (1h11m37.979581727s) [noise=true , address=79.127.201.215:4358 , piece_id=RRICN3R6RRF2MIBZM2ZXIXI6NG2QKRAAMCTZ5ZUQOQBRHHHWANDQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: <nil> Jul 26 23:03:37.964568820 (1h11m53.7455022s) [noise=true , address=77.90.31.11:4571 , piece_id=QVQ45R3X7IMH22OC3MTV3MJC6YQSCVWNEEZB3266TPDSB2NDZAKQ, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 159.223.143.39:42210->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:37.965835006 (1h11m53.746768388s) [noise=true , address=77.90.31.15:4727 , piece_id=6E7RVTU3GRXXXIOJWRIXW36JASGTOOYJ7JTG5VUUGGNJTJQKIRHA, seg_pos={0 0}, upload=1100]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 159.223.143.39:36482->77.90.31.15:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:37.967314376 (1h11m53.74824777s) [seg_pos={0 0}, upload=1100]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:03:37.994728808 (1h11m53.775662191s) [upload=1100]: Done uploading segments... Jul 26 23:03:37.994772690 (1h11m53.775706073s) [upload=1100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.033076235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........X................................................] duration: 402.013851ms FAIL (2025-07-26 23:03:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 402.051683ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 182.901955ms FAIL (2025-07-26 23:03:14 UTC) orbiter warning: uplink: too many requests Duration: 182.930296ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 203.446951ms FAIL (2025-07-26 23:02:51 UTC) orbiter warning: uplink: too many requests Duration: 203.519434ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X................X........................X..] duration: 1.290697241s FAIL (2025-07-26 23:02:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.29072857s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X........X.X.X...XX.................X.X..........XX...X...X.] duration: 13.715032586s FAIL (2025-07-26 23:02:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.715054331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXX] duration: 1m0.031115054s FAIL (2025-07-26 23:02:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 1083]: Done begin upload segment. Jul 26 23:01:37.955259301 (1h9m53.736192688s) [noise=true , address=77.90.31.10:4534 , piece_id=UG5WDILADUZ2YXHCPFJHLVFGJG7HGNIVWDNZRR6TQSYH6PUJJTCA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.955276692 (1h9m53.736210081s) [noise=true , address=77.90.31.16:4736 , piece_id=NHAJJTMG5PERTFZR6WD4YQD2ZQ5TECB3LPVGT76UXHDXU7KU2FPQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.955749101 (1h9m53.736682469s) [noise=true , address=77.90.31.11:4585 , piece_id=BXT66X4AXVPI5GUJCIJFLPG4WTTCUBZHL2ENNBY3AEF34CKNIONA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.955798941 (1h9m53.736732328s) [noise=true , address=79.127.205.233:4647 , piece_id=TA6V2BTP63SCNHEEHREUAUFSAZDAFK7PTHBDF356XUAEYOJSTVTQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956178165 (1h9m53.737111553s) [noise=true , address=156.146.43.225:4434 , piece_id=7SCYRR6HLFEJ3CTGVHUAKUPHETXFN25JO5IB7UCFSKKSB2T2OPSQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956375061 (1h9m53.737308441s) [noise=true , address=77.90.31.15:4729 , piece_id=NLOGILCQR5KWDQKLMMTD7SNJE554ETO6NY3FB6ZD24FLVQB2LPDQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956592618 (1h9m53.737525985s) [noise=true , address=156.146.43.225:4449 , piece_id=XFJXEWDXZX6JBXWMCK6LNTFESOGRGNWIFSQ744BQOSF5XU4RC2FA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956829052 (1h9m53.737762434s) [noise=true , address=79.127.205.245:4225 , piece_id=Q72CVB6AKSP2JZ4TNDM2ALOSJNN2I2MREPP7QCWOJRFKVGWMNSIA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.956975793 (1h9m53.737909194s) [noise=true , address=79.127.205.245:4241 , piece_id=R2C7HZUI6O2SSM6CRKML3CQG73BHISVD375VPG64MPWHY4NYIQRA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.957269512 (1h9m53.738202893s) [noise=true , address=77.90.31.17:4800 , piece_id=STTGDD2DRKHT4JQYSTJSJR6ZXWGK3HJK6OZWYVF5AJD6QDUSTPPA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.957405477 (1h9m53.738338868s) [noise=true , address=79.127.219.48:4090 , piece_id=IEQCATMUBUYU4APPUPSE6V4WQ4WETSL63HJUAQUDMX6PG7XXKHSA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.957675299 (1h9m53.738608682s) [noise=true , address=77.90.31.19:4871 , piece_id=IPV34RFOGY75HQ5DQPGGDAKXOZGAMOVFZO6KL72HB4I4MU3ZNXZA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.957750452 (1h9m53.738683852s) [noise=true , address=79.127.205.246:4278 , piece_id=4KZ3G6HYD22GA7B7TGTGDWQXONCIEZST7EH7MX4FR6ELBECA6TRQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.958075199 (1h9m53.739008578s) [noise=true , address=79.127.205.235:4690 , piece_id=7T5XI6OGOLG2NQWUUJZJPGLB4GXS475FFGK6BPUVMMNTPBEM2BZA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.958107878 (1h9m53.739041257s) [noise=true , address=79.127.205.234:4601 , piece_id=UNCAAY2JYM2KQ3ROLPPDNWC2UJ6J53LAOF3X67CO4ZH72VQWX3NQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.958492712 (1h9m53.73942609s) [noise=true , address=79.127.219.38:4009 , piece_id=TNR6B2SXPHPPXEIEGMDOCUZHAQSZW3QPI2DYSXOU67FW2SR7TGJA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.958944171 (1h9m53.739877554s) [noise=true , address=77.90.31.12:4611 , piece_id=RBE2BQV5TZY5R73MPIHPHCP7HK3KPKLF6VUT66VSQPXK6T4V3UAA, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:37.959452651 (1h9m53.740386031s) [noise=true , address=77.90.31.11:4572 , piece_id=MLCI7LJHVLOQHRGBFDGJUSZULLTTELKN7R5OV5VXOTOBPJPR35OQ, seg_pos={0 0}, upload=1083]: Uploading piece... Jul 26 23:01:38.005878674 (1h9m53.786812069s) [noise=true , address=77.90.31.19:4871 , piece_id=IPV34RFOGY75HQ5DQPGGDAKXOZGAMOVFZO6KL72HB4I4MU3ZNXZA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:38.026684894 (1h9m53.807618274s) [noise=true , address=79.127.219.48:4090 , piece_id=IEQCATMUBUYU4APPUPSE6V4WQ4WETSL63HJUAQUDMX6PG7XXKHSA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:38.055552473 (1h9m53.836485838s) [upload=1083]: Next returned nil segment Jul 26 23:01:38.055583547 (1h9m53.836516934s) [upload=1083]: Waiting for error group managing segments... Jul 26 23:01:38.147086153 (1h9m53.928019518s) [noise=true , address=79.127.219.38:4009 , piece_id=TNR6B2SXPHPPXEIEGMDOCUZHAQSZW3QPI2DYSXOU67FW2SR7TGJA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.441336814 (1h9m54.222270203s) [noise=true , address=156.146.43.225:4449 , piece_id=XFJXEWDXZX6JBXWMCK6LNTFESOGRGNWIFSQ744BQOSF5XU4RC2FA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:38.595971390 (1h9m54.376904755s) [noise=true , address=79.127.205.246:4278 , piece_id=4KZ3G6HYD22GA7B7TGTGDWQXONCIEZST7EH7MX4FR6ELBECA6TRQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.606153946 (1h9m54.38708731s) [noise=true , address=79.127.205.245:4225 , piece_id=Q72CVB6AKSP2JZ4TNDM2ALOSJNN2I2MREPP7QCWOJRFKVGWMNSIA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.624838426 (1h9m54.405771805s) [noise=true , address=79.127.205.235:4690 , piece_id=7T5XI6OGOLG2NQWUUJZJPGLB4GXS475FFGK6BPUVMMNTPBEM2BZA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.642426432 (1h9m54.423359804s) [noise=true , address=79.127.205.233:4647 , piece_id=TA6V2BTP63SCNHEEHREUAUFSAZDAFK7PTHBDF356XUAEYOJSTVTQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.657972779 (1h9m54.438906149s) [noise=true , address=79.127.205.234:4601 , piece_id=UNCAAY2JYM2KQ3ROLPPDNWC2UJ6J53LAOF3X67CO4ZH72VQWX3NQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:38.658056837 (1h9m54.438990215s) [noise=true , address=79.127.205.245:4241 , piece_id=R2C7HZUI6O2SSM6CRKML3CQG73BHISVD375VPG64MPWHY4NYIQRA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:40.030667657 (1h9m55.811601026s) [noise=true , address=77.90.31.12:4611 , piece_id=RBE2BQV5TZY5R73MPIHPHCP7HK3KPKLF6VUT66VSQPXK6T4V3UAA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:41.700806954 (1h9m57.481740327s) [noise=true , address=156.146.43.225:4434 , piece_id=7SCYRR6HLFEJ3CTGVHUAKUPHETXFN25JO5IB7UCFSKKSB2T2OPSQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: <nil> Jul 26 23:01:50.720954465 (1h10m6.501887844s) [noise=true , address=77.90.31.17:4800 , piece_id=STTGDD2DRKHT4JQYSTJSJR6ZXWGK3HJK6OZWYVF5AJD6QDUSTPPA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:37.865994557 (1h10m53.64692795s) [noise=true , address=77.90.31.11:4585 , piece_id=BXT66X4AXVPI5GUJCIJFLPG4WTTCUBZHL2ENNBY3AEF34CKNIONA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: rpc: dial tcp 77.90.31.11:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:37.866189732 (1h10m53.647123117s) [noise=true , address=77.90.31.15:4729 , piece_id=NLOGILCQR5KWDQKLMMTD7SNJE554ETO6NY3FB6ZD24FLVQB2LPDQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: rpc: dial tcp 77.90.31.15:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:37.866275913 (1h10m53.647209292s) [noise=true , address=77.90.31.11:4572 , piece_id=MLCI7LJHVLOQHRGBFDGJUSZULLTTELKN7R5OV5VXOTOBPJPR35OQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: write tcp 159.223.143.39:32774->77.90.31.11:4572: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:37.866381094 (1h10m53.647314486s) [noise=true , address=77.90.31.16:4736 , piece_id=NHAJJTMG5PERTFZR6WD4YQD2ZQ5TECB3LPVGT76UXHDXU7KU2FPQ, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: write tcp 159.223.143.39:50350->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:37.866439679 (1h10m53.647373062s) [noise=true , address=77.90.31.10:4534 , piece_id=UG5WDILADUZ2YXHCPFJHLVFGJG7HGNIVWDNZRR6TQSYH6PUJJTCA, seg_pos={0 0}, upload=1083]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: read tcp 159.223.143.39:41702->77.90.31.10:4534: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:02:37.866479299 (1h10m53.647412688s) [seg_pos={0 0}, upload=1083]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:02:37.892659719 (1h10m53.673593109s) [upload=1083]: Done uploading segments... Jul 26 23:02:37.892706533 (1h10m53.673639916s) [upload=1083]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.031143897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X........X...............................................] duration: 620.592561ms FAIL (2025-07-26 23:02:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 620.6215ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.X........X.X.X...XX.................X.X..........XX...X...X] duration: 1m0.000547833s FAIL (2025-07-26 23:02:21 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00058487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............X................X........................X.] duration: 466.03655ms FAIL (2025-07-26 23:02:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 466.067961ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 194.636927ms FAIL (2025-07-26 23:02:06 UTC) orbiter warning: uplink: too many requests Duration: 194.668572ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X........X..............................................] duration: 553.132114ms FAIL (2025-07-26 23:02:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 553.168331ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............X................X........................X] duration: 30.03288738s FAIL (2025-07-26 23:01:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.03294042s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXX] duration: 1m0.027591794s FAIL (2025-07-26 23:01:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== s) [upload=1048]: Done uploading segments... Jul 26 23:00:28.609366837 (1h8m44.390300223s) [upload=1048]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Jul 26 23:00:28.779559960 (1h8m44.560493328s) [upload=1066]: Starting upload Jul 26 23:00:28.779653135 (1h8m44.560586515s) [upload=1066]: Done starting upload Jul 26 23:00:28.779750366 (1h8m44.560683746s) [upload=1066]: Uploading segments... Jul 26 23:00:28.779923494 (1h8m44.560856871s) [upload=1066]: Got next segment. Inline: false Jul 26 23:00:28.779977957 (1h8m44.560911319s) [upload=1066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:28.779987601 (1h8m44.560921051s) [upload=1066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:28.876762903 (1h8m44.657696341s) [seg_pos={0 0}, upload=1066]: Begin upload segment... Jul 26 23:00:28.876845194 (1h8m44.657778585s) [seg_pos={0 0}, upload=1066]: Done begin upload segment. Jul 26 23:00:28.876888922 (1h8m44.65782231s) [noise=true , address=77.90.31.19:4871 , piece_id=GF7VPASF5UYDVYIZPBYB6B3X7ZVT6NE5HBBEGBFZS2PC6I4BKY4A, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.877532636 (1h8m44.658466046s) [noise=true , address=79.127.205.234:4603 , piece_id=GMJX6KUOV4T6FLFJDBWOQAZKY4IRQBJAKIBI6RJHERJ7HB3LVG4Q, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.878032452 (1h8m44.658965841s) [noise=true , address=45.91.116.5:4310 , piece_id=4I3OEX4PZFA5S2WL2RQ3AQXB6SXVBBFPYWSWZ4GAQONNEENKPHYQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.878520889 (1h8m44.65945434s) [noise=true , address=77.90.31.12:4601 , piece_id=KB4MRV7ZA3HR2AFVCPLHVU3VCPUDKKOG6XZASYNKZGHQDD3HMTAQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.878798467 (1h8m44.659731852s) [noise=true , address=45.91.116.11:4121 , piece_id=PAE6BE67CPC5WSBULNMGXI3G254M43SQKTXWMYISJILZYTPKCOZQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.882673502 (1h8m44.663606868s) [noise=true , address=79.127.219.38:4006 , piece_id=QRZQFXG53KTASLNYDPJXAUE5RUM36AM4DSBFQZ33FIWLTZXIAQMQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.883549394 (1h8m44.664482782s) [noise=true , address=79.127.205.235:4657 , piece_id=YDWT72JHEPG6EGIG2HHTV4SSBDVWN6XMV2VBOZZ35MRXKHZ6VHZQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.884146773 (1h8m44.665080153s) [noise=true , address=79.127.219.38:4005 , piece_id=AHJN4QNVYHEH27T2BNWCLDUZ4NURHDPAVYMGXS275YLZZMNRYRYA, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.884904311 (1h8m44.665837698s) [noise=true , address=77.90.31.21:4897 , piece_id=77JABZES2DNPGERLELV6VYHSO3CENV4J6BF6WG35IKWOJ6S64PQQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.886476794 (1h8m44.667410183s) [noise=true , address=79.127.205.246:4278 , piece_id=BPCV4TRQBW26MCFOQGDOLS4RMG4GIYHUWNNW25GIQ7BMKDDHZUMQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.887056314 (1h8m44.667989703s) [noise=true , address=79.127.205.233:4647 , piece_id=OGOR3NQX7KGYY36IGCXDIW4HSEJYE6OQNAOOJOU3MUA2NUVYOWRQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.887597951 (1h8m44.66853134s) [noise=true , address=45.91.116.14:4194 , piece_id=MV65HCFWZZWTISYGQ3CN3MJW65PKTPHVKZMUZGTY5TTWYP6O7NTQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.888162033 (1h8m44.669095421s) [noise=true , address=45.91.116.14:4222 , piece_id=L6AUWAS3LFK45PVRLOQYAGE5A6TODMQI6CC4UET4KZH3BN72JQLA, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.888924172 (1h8m44.669857557s) [noise=true , address=45.91.116.13:4175 , piece_id=3E2T7QCWKLSKF35CUNKRQGOEE6TPB4M6SUJ6LL4BD7Q6SHCWH7SQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.889840118 (1h8m44.670773504s) [noise=true , address=79.127.219.48:4090 , piece_id=YSBAII4FSIBGT27EJUE6GYD44NVPM5QT4HIMRCMP2XD7Z2ZEESRQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.890474302 (1h8m44.671407703s) [noise=true , address=77.90.31.13:4644 , piece_id=OOABN4245V6AO2LIZFGMMPAWLZCY7GCGFNUO22QZI46RFGIVE4CA, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.891354729 (1h8m44.67228812s) [noise=true , address=64.49.13.170:4719 , piece_id=KITJUI5UO6AS4FPLDUUI7IBDSGLKMVUE7ASDEXWRI34U6GYP7NPQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.892054191 (1h8m44.67298758s) [noise=true , address=64.49.13.139:4478 , piece_id=MNQFARQXMP532LW3MXX7ORIQJAD7RFYD6TVTZ5QFNRMWMYECAYBQ, seg_pos={0 0}, upload=1066]: Uploading piece... Jul 26 23:00:28.958309374 (1h8m44.739242742s) [noise=true , address=79.127.219.38:4006 , piece_id=QRZQFXG53KTASLNYDPJXAUE5RUM36AM4DSBFQZ33FIWLTZXIAQMQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:18qiLeQCqaa6Sa898QMfKEZo68t1JKbDMk4ixJjTzcFgL7aes3, address:79.127.219.38:4006): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:29.060212463 (1h8m44.841145833s) [upload=1066]: Next returned nil segment Jul 26 23:00:29.060380258 (1h8m44.841313646s) [upload=1066]: Waiting for error group managing segments... Jul 26 23:00:29.265127244 (1h8m45.046060624s) [noise=true , address=77.90.31.12:4601 , piece_id=KB4MRV7ZA3HR2AFVCPLHVU3VCPUDKKOG6XZASYNKZGHQDD3HMTAQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:29.574612772 (1h8m45.355546155s) [noise=true , address=79.127.205.233:4647 , piece_id=OGOR3NQX7KGYY36IGCXDIW4HSEJYE6OQNAOOJOU3MUA2NUVYOWRQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:29.592311070 (1h8m45.373244454s) [noise=true , address=79.127.205.246:4278 , piece_id=BPCV4TRQBW26MCFOQGDOLS4RMG4GIYHUWNNW25GIQ7BMKDDHZUMQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:29.603602389 (1h8m45.384535777s) [noise=true , address=79.127.205.234:4603 , piece_id=GMJX6KUOV4T6FLFJDBWOQAZKY4IRQBJAKIBI6RJHERJ7HB3LVG4Q, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:29.733428276 (1h8m45.514361647s) [noise=true , address=79.127.205.235:4657 , piece_id=YDWT72JHEPG6EGIG2HHTV4SSBDVWN6XMV2VBOZZ35MRXKHZ6VHZQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:30.299891512 (1h8m46.080824891s) [noise=true , address=45.91.116.14:4194 , piece_id=MV65HCFWZZWTISYGQ3CN3MJW65PKTPHVKZMUZGTY5TTWYP6O7NTQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:30.302944110 (1h8m46.083877493s) [noise=true , address=45.91.116.5:4310 , piece_id=4I3OEX4PZFA5S2WL2RQ3AQXB6SXVBBFPYWSWZ4GAQONNEENKPHYQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:30.487640710 (1h8m46.268574086s) [noise=true , address=64.49.13.170:4719 , piece_id=KITJUI5UO6AS4FPLDUUI7IBDSGLKMVUE7ASDEXWRI34U6GYP7NPQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:31.985030848 (1h8m47.765964225s) [noise=true , address=77.90.31.19:4871 , piece_id=GF7VPASF5UYDVYIZPBYB6B3X7ZVT6NE5HBBEGBFZS2PC6I4BKY4A, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:32.180639051 (1h8m47.961572424s) [noise=true , address=45.91.116.11:4121 , piece_id=PAE6BE67CPC5WSBULNMGXI3G254M43SQKTXWMYISJILZYTPKCOZQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:35.193917160 (1h8m50.974850528s) [noise=true , address=79.127.219.48:4090 , piece_id=YSBAII4FSIBGT27EJUE6GYD44NVPM5QT4HIMRCMP2XD7Z2ZEESRQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:37.970649671 (1h8m53.751583037s) [noise=true , address=79.127.219.38:4005 , piece_id=AHJN4QNVYHEH27T2BNWCLDUZ4NURHDPAVYMGXS275YLZZMNRYRYA, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:00:37.973852054 (1h8m53.754785434s) [noise=true , address=45.91.116.13:4175 , piece_id=3E2T7QCWKLSKF35CUNKRQGOEE6TPB4M6SUJ6LL4BD7Q6SHCWH7SQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:42.795986444 (1h8m58.576919825s) [noise=true , address=45.91.116.14:4222 , piece_id=L6AUWAS3LFK45PVRLOQYAGE5A6TODMQI6CC4UET4KZH3BN72JQLA, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:47.116949998 (1h9m2.897883378s) [noise=true , address=64.49.13.139:4478 , piece_id=MNQFARQXMP532LW3MXX7ORIQJAD7RFYD6TVTZ5QFNRMWMYECAYBQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: <nil> Jul 26 23:01:28.707826424 (1h9m44.488759804s) [noise=true , address=77.90.31.13:4644 , piece_id=OOABN4245V6AO2LIZFGMMPAWLZCY7GCGFNUO22QZI46RFGIVE4CA, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: rpc: dial tcp 77.90.31.13:4644: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:28.707936095 (1h9m44.488869474s) [noise=true , address=77.90.31.21:4897 , piece_id=77JABZES2DNPGERLELV6VYHSO3CENV4J6BF6WG35IKWOJ6S64PQQ, seg_pos={0 0}, upload=1066]: Done uploading piece. err: ecclient: upload failed (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, address:77.90.31.21:4897): protocol: write tcp 159.223.143.39:38006->77.90.31.21:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:28.707958133 (1h9m44.48889151s) [seg_pos={0 0}, upload=1066]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:01:28.733358789 (1h9m44.514292169s) [upload=1066]: Done uploading segments... Jul 26 23:01:28.733388630 (1h9m44.514322012s) [upload=1066]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.027630659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X....X........X.............................................] duration: 385.283083ms FAIL (2025-07-26 23:01:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.314746ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 193.425992ms FAIL (2025-07-26 23:01:00 UTC) orbiter warning: uplink: too many requests Duration: 193.595142ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X........X.X.X...XX.................X.X..........XX...X...] duration: 24.171562333s FAIL (2025-07-26 23:00:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.171628227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XX] duration: 1m0.026231565s FAIL (2025-07-26 23:00:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:59:28.632785432 (1h7m44.413718797s) [upload=1048]: Starting upload Jul 26 22:59:28.632852774 (1h7m44.413786151s) [upload=1048]: Done starting upload Jul 26 22:59:28.632966996 (1h7m44.413900373s) [upload=1048]: Uploading segments... Jul 26 22:59:28.633135249 (1h7m44.414068619s) [upload=1048]: Got next segment. Inline: false Jul 26 22:59:28.633179735 (1h7m44.414113103s) [upload=1048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:59:28.633185044 (1h7m44.414118415s) [upload=1048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:59:28.675174091 (1h7m44.456107478s) [seg_pos={0 0}, upload=1048]: Begin upload segment... Jul 26 22:59:28.675319246 (1h7m44.456252633s) [noise=true , address=79.127.205.233:4640 , piece_id=M33KSG4V3CPE3DDPORB353USNKP5K2GZK23JQVINF3QAVLP42RHA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.675985706 (1h7m44.456919104s) [noise=true , address=79.127.205.246:4273 , piece_id=EYG55YHPP4INLILEI53J4D76VE4DIGNUVHW46YKOJTH23Z2VRT2A, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.676703310 (1h7m44.457636714s) [noise=true , address=77.90.31.17:4800 , piece_id=DJ4TGCEYHNLBLLDMKHXQOTKGWHAC5GVQCRF6FJFAQW5JQQSC5JQQ, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.677403064 (1h7m44.458336451s) [noise=true , address=77.90.31.12:4608 , piece_id=D75VON4TP5PK24242JAXN33CEXTUP5GAYCCX236H5TCZENX6WWWA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.677889334 (1h7m44.458822723s) [noise=true , address=77.90.31.11:4574 , piece_id=5S3J6PA7L5SCUVFX7MEK7APPDK6O4C547VU6TUGOS4QR3DSMFYMA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.679977009 (1h7m44.460910389s) [seg_pos={0 0}, upload=1048]: Done begin upload segment. Jul 26 22:59:28.679998989 (1h7m44.460932355s) [noise=true , address=79.127.219.38:4006 , piece_id=R2Z4K2KZF64ASWD7XGF4GGCR5PM65KYKW64NV73ZBXIHDN7MFNVA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.680624581 (1h7m44.461557974s) [noise=true , address=79.127.205.245:4236 , piece_id=2ELWIPVPX4FHJMHUVC4E2FGME6CJGUQZXRNQYH4IKM7F7ZGKUB6Q, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.681491378 (1h7m44.462424741s) [noise=true , address=64.49.13.140:4514 , piece_id=QNOHYVKSOXOXFLEKZFZDQ3EUMZTOJ5YSLRKS5UXLAUJELKWMAMSA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.682360718 (1h7m44.463294106s) [noise=true , address=79.127.205.246:4262 , piece_id=4WJH3ALU6LIKR67AZOC6HH43VYRJCXNIYIJS3T6KFH4M7XFGHMQA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.684909750 (1h7m44.465843119s) [noise=true , address=79.127.219.47:4048 , piece_id=NBYL4UPTLCLBB47SIHTEQ4PIM4K3SOA64RYEPM56VTT72QRLRTNQ, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.687832780 (1h7m44.468766169s) [noise=true , address=79.127.205.245:4243 , piece_id=GGG2MCOQXAW3FYALXOTKL4AJJ2QCJ6LYRKCQM2UIXF5MORJCKIAA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.692412427 (1h7m44.473345791s) [noise=true , address=77.90.31.10:4538 , piece_id=5ZX4B5Z7LDJY3SMCGHBJ5RSHF26FGFPU6JV7IJU72HG4EIC3433Q, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.695376024 (1h7m44.476309418s) [noise=true , address=79.127.205.235:4679 , piece_id=JUUD3SGKUG2IF3VDOUFRTD4Z7AUJ5K3GXDOIL7X4ULSI5SJG63QA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.700353033 (1h7m44.481286417s) [noise=true , address=79.127.205.235:4657 , piece_id=QVDH35DXHQLCPTB3D4ER4IW755UKEDBPSV6IEO3RWDVNCBCJH54A, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.702827442 (1h7m44.483760832s) [noise=true , address=77.90.31.10:4531 , piece_id=AAXYJAOQQP6KDLXKPPKTCKBWS5NBD7QUC4F6ONAXCJGMJN5UI7QQ, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.704352429 (1h7m44.485285814s) [noise=true , address=45.91.116.14:4194 , piece_id=VR5IKMXYQDAERLGFQWLHU36SN32RTOJMECAYUWMRSEUOPFRMGL2A, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.705071967 (1h7m44.486005351s) [noise=true , address=77.90.31.19:4867 , piece_id=HIONQWFHD6XCEJ4M2HGAYATF7JYTX3SD4DO3LACZE4FEFKQZ2XKQ, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.707684100 (1h7m44.488617489s) [noise=true , address=79.127.220.97:4361 , piece_id=LQH5P74Q2FFC66W4VVRDK5ULWYF3ALKD6DDQ46CFGN2MBKMTKRQA, seg_pos={0 0}, upload=1048]: Uploading piece... Jul 26 22:59:28.720993283 (1h7m44.501926666s) [noise=true , address=79.127.219.47:4048 , piece_id=NBYL4UPTLCLBB47SIHTEQ4PIM4K3SOA64RYEPM56VTT72QRLRTNQ, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:28.731535325 (1h7m44.512468692s) [noise=true , address=79.127.219.38:4006 , piece_id=R2Z4K2KZF64ASWD7XGF4GGCR5PM65KYKW64NV73ZBXIHDN7MFNVA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:18qiLeQCqaa6Sa898QMfKEZo68t1JKbDMk4ixJjTzcFgL7aes3, address:79.127.219.38:4006): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:28.732993931 (1h7m44.513927315s) [noise=true , address=77.90.31.10:4531 , piece_id=AAXYJAOQQP6KDLXKPPKTCKBWS5NBD7QUC4F6ONAXCJGMJN5UI7QQ, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:28.759175430 (1h7m44.540108813s) [upload=1048]: Next returned nil segment Jul 26 22:59:28.759197828 (1h7m44.540131215s) [upload=1048]: Waiting for error group managing segments... Jul 26 22:59:28.773102137 (1h7m44.554035507s) [noise=true , address=77.90.31.12:4608 , piece_id=D75VON4TP5PK24242JAXN33CEXTUP5GAYCCX236H5TCZENX6WWWA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.087575082 (1h7m44.868508449s) [noise=true , address=79.127.220.97:4361 , piece_id=LQH5P74Q2FFC66W4VVRDK5ULWYF3ALKD6DDQ46CFGN2MBKMTKRQA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:29.157137363 (1h7m44.938070756s) [noise=true , address=64.49.13.140:4514 , piece_id=QNOHYVKSOXOXFLEKZFZDQ3EUMZTOJ5YSLRKS5UXLAUJELKWMAMSA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:29.276319050 (1h7m45.057252413s) [noise=true , address=79.127.205.233:4640 , piece_id=M33KSG4V3CPE3DDPORB353USNKP5K2GZK23JQVINF3QAVLP42RHA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.294250921 (1h7m45.075184291s) [noise=true , address=79.127.205.235:4679 , piece_id=JUUD3SGKUG2IF3VDOUFRTD4Z7AUJ5K3GXDOIL7X4ULSI5SJG63QA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.304970209 (1h7m45.085903583s) [noise=true , address=79.127.205.245:4243 , piece_id=GGG2MCOQXAW3FYALXOTKL4AJJ2QCJ6LYRKCQM2UIXF5MORJCKIAA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.306288359 (1h7m45.087221736s) [noise=true , address=79.127.205.246:4262 , piece_id=4WJH3ALU6LIKR67AZOC6HH43VYRJCXNIYIJS3T6KFH4M7XFGHMQA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.334484858 (1h7m45.115418228s) [noise=true , address=79.127.205.246:4273 , piece_id=EYG55YHPP4INLILEI53J4D76VE4DIGNUVHW46YKOJTH23Z2VRT2A, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.342984491 (1h7m45.123917861s) [noise=true , address=79.127.205.245:4236 , piece_id=2ELWIPVPX4FHJMHUVC4E2FGME6CJGUQZXRNQYH4IKM7F7ZGKUB6Q, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.438509939 (1h7m45.219443316s) [noise=true , address=79.127.205.235:4657 , piece_id=QVDH35DXHQLCPTB3D4ER4IW755UKEDBPSV6IEO3RWDVNCBCJH54A, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.797630748 (1h7m45.578564125s) [noise=true , address=77.90.31.17:4800 , piece_id=DJ4TGCEYHNLBLLDMKHXQOTKGWHAC5GVQCRF6FJFAQW5JQQSC5JQQ, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:29.956952019 (1h7m45.73788539s) [noise=true , address=77.90.31.10:4538 , piece_id=5ZX4B5Z7LDJY3SMCGHBJ5RSHF26FGFPU6JV7IJU72HG4EIC3433Q, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 22:59:32.829679251 (1h7m48.61061262s) [noise=true , address=45.91.116.14:4194 , piece_id=VR5IKMXYQDAERLGFQWLHU36SN32RTOJMECAYUWMRSEUOPFRMGL2A, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 23:00:14.856239221 (1h8m30.637172585s) [noise=true , address=77.90.31.19:4867 , piece_id=HIONQWFHD6XCEJ4M2HGAYATF7JYTX3SD4DO3LACZE4FEFKQZ2XKQ, seg_pos={0 0}, upload=1048]: Done uploading piece. err: <nil> Jul 26 23:00:28.586262741 (1h8m44.367196115s) [noise=true , address=77.90.31.11:4574 , piece_id=5S3J6PA7L5SCUVFX7MEK7APPDK6O4C547VU6TUGOS4QR3DSMFYMA, seg_pos={0 0}, upload=1048]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:44748->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:28.586295677 (1h8m44.367229066s) [seg_pos={0 0}, upload=1048]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:00:28.609336109 (1h8m44.390269492s) [upload=1048]: Done uploading segments... Jul 26 23:00:28.609366837 (1h8m44.390300223s) [upload=1048]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.026255195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....X........X............................................] duration: 634.099618ms FAIL (2025-07-26 23:00:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 634.203413ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X........X.X.X...XX.................X.X..........XX...X..] duration: 11.755287686s FAIL (2025-07-26 23:00:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.755314408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.............X................X........................] duration: 781.075341ms FAIL (2025-07-26 23:00:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 781.113459ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [....X........X.X.X...XX.................X.X..........XX...X.] duration: 7.129169372s FAIL (2025-07-26 22:59:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.129185492s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)