Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX.X.XXX.X..X...............................................] duration: 2.466852023s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:54:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.466886067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.109015592s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:54:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.109047007s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.......] duration: 2.761422862s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:54:11 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.76144276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX.X.XXX.X..X..............................................] duration: 3.621166969s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:53:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.621219841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 4.97313619s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:53:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.973204835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X......] duration: 3.578046131s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:53:31 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578079072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XX.X.XXX.X..X.............................................] duration: 3.871792646s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:53:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.871844262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...XX.X.XXX.X..X............................................] duration: 4.362800108s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:53:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.362930791s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.....] duration: 4.165319255s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:52:48 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.165342674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.795096363s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:52:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.795151869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....] duration: 3.52836284s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:52:31 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528384781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...XX.X.XXX.X..X...........................................] duration: 4.660317862s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:52:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.660376754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.082675923s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:51:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.082748805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...] duration: 4.408964404s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:51:37 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.408995997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......X...................................................] duration: 5.237261124s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:51:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.237334781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX...XX.X.XXX.X..X..........................................] duration: 4.436979315s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:51:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.437009242s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..] duration: 4.318657162s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:50:59 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.318705077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 3.939329812s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:50:36 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.939362991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......X..................................................] duration: 5.996890138s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:50:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.996940815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXX...XX.X.XXX.X..X.........................................] duration: 5.456486409s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:50:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.456514353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXX...XX.X.XXX.X..X........................................] duration: 5.087616634s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:49:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.087663969s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.001462444s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:49:44 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001493084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......X.................................................] duration: 4.162696898s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:49:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.162720437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 4.503472136s trace: https://jaeger.dev.storj.io/trace/2f7135ea0b1f08a0 FAIL (2026-03-18 20:49:03 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.503644002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001317933s trace: https://jaeger.dev.storj.io/trace/2f7135ea0b1f08a0 FAIL (2026-03-18 20:48:41 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001345043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXX...XX.X.XXX.X..X.......................................] duration: 4.56360347s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:48:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.563634646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXX...XX.X.XXX.X..X......................................] duration: 6.12868596s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:48:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.128809677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......X................................................] duration: 7.767833888s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:48:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.767897051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.00106994s trace: https://jaeger.dev.storj.io/trace/3af8c44bddd0476d FAIL (2026-03-18 20:47:46 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001094049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.......X...............................................] duration: 8.559114002s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:47:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.559144381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXXXXX...XX.X.XXX.X..X.....................................] duration: 4.767066463s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:47:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.767083293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X....X.......X..............................................] duration: 8.578482741s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:47:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.578608062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002035039s trace: https://jaeger.dev.storj.io/trace/5039559111b3eb88 FAIL (2026-03-18 20:46:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002064027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....X.......X.............................................] duration: 9.469282381s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:46:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.46932945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....X.......X............................................] duration: 9.832519897s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:46:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.832571608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XXXXXX...XX.X.XXX.X..X....................................] duration: 8.755164512s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:46:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.755190844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.0007782s trace: https://jaeger.dev.storj.io/trace/4c70ad8a0a37332d FAIL (2026-03-18 20:45:59 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000801208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.XXXXXX...XX.X.XXX.X..X...................................] duration: 8.756470797s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:45:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.756502233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....X.......X...........................................] duration: 9.745138864s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:45:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.745241979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....X.......X..........................................] duration: 9.666992533s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:45:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.66702338s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001338302s trace: https://jaeger.dev.storj.io/trace/2e9674795d579581 FAIL (2026-03-18 20:45:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001355398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX.XXXXXX...XX.X.XXX.X..X..................................] duration: 8.82847127s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:45:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.82850314s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002338767s trace: https://jaeger.dev.storj.io/trace/70cf6f6b635be038 FAIL (2026-03-18 20:45:05 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002364608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.989353372s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:44:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.9894583s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X....X.......X.........................................] duration: 8.164805152s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:44:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.1648337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.966631595s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.966742494s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXX.XXXXXX...XX.X.XXX.X..X.................................] duration: 8.769651677s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:44:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.769699753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....X....X.......X........................................] duration: 9.928734331s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:44:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.928766578s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX................XXXXXXXXXXXXXXXXXXXXXX] duration: 5.001483868s trace: https://jaeger.dev.storj.io/trace/3009bcfb01ce93ac FAIL (2026-03-18 20:44:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001533464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X....X....X.......X.......................................] duration: 9.546091256s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:43:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.546196176s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)