Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X...................] duration: 9.273998526s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:34:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.274027653s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...X.........X....X....X.......X..........................] duration: 9.509328282s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:33:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.509356317s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....X........X............................................] duration: 8.610322789s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:33:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.610370769s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................XXXXXXXXX] duration: 5.000693921s trace: https://jaeger.dev.storj.io/trace/39a4e6884758646 FAIL (2026-03-18 20:33:21 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00078264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X..................] duration: 8.683281728s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:33:04 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.683326866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....X........X...........................................] duration: 10.135417765s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:32:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.135460317s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...X.........X....X....X.......X.........................] duration: 9.556072546s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:32:34 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.556100049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................XXXXXXXX] duration: 5.001930087s trace: https://jaeger.dev.storj.io/trace/6fc07624bfc80ec8 FAIL (2026-03-18 20:32:27 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00196084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X.................] duration: 8.565610129s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:32:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.565692993s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................XXXXXXX] duration: 5.001582757s trace: https://jaeger.dev.storj.io/trace/14d3da8de6c50a42 FAIL (2026-03-18 20:32:05 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002006439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X................] duration: 8.287573974s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:32:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.287621623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..X....X........X..........................................] duration: 8.955180423s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:31:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.955268586s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................XXXXXX] duration: 5.001092472s trace: https://jaeger.dev.storj.io/trace/36abdfb9883a16ed FAIL (2026-03-18 20:31:37 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001121284s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...X.........X....X....X.......X........................] duration: 6.39286127s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:31:29 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.392886734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X...............] duration: 8.182173524s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:31:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.182204698s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX..X....X........X.........................................] duration: 9.075153198s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:31:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.075181562s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................XXXXX] duration: 5.000918427s trace: https://jaeger.dev.storj.io/trace/3219754200e8957c FAIL (2026-03-18 20:31:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000941611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X...X.........X....X....X.......X.......................] duration: 8.241289234s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:31:04 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.241320586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X..............] duration: 6.696001948s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:30:41 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.696022056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX..X....X........X........................................] duration: 8.109049976s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:30:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.109185095s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................XXXX] duration: 5.002170767s trace: https://jaeger.dev.storj.io/trace/450c6cfd22ba7851 FAIL (2026-03-18 20:30:29 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002196734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X...X.........X....X....X.......X......................] duration: 6.881569648s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:30:16 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.881605678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX..X....X........X.......................................] duration: 7.897169515s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:29:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.897195718s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................XXX] duration: 5.001725106s trace: https://jaeger.dev.storj.io/trace/3d8455f3f0bfb8bb FAIL (2026-03-18 20:29:48 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001753913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X.............] duration: 7.035406052s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:29:38 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.035649376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X...X.........X....X....X.......X.....................] duration: 5.600907003s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:29:36 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.600935496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................XX] duration: 5.001670332s trace: https://jaeger.dev.storj.io/trace/1f8f52a52d3d67ba FAIL (2026-03-18 20:29:32 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001764616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX..X....X........X......................................] duration: 7.341668046s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:29:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.341713216s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X............] duration: 6.581443831s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:29:23 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.581476596s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................X] duration: 5.001158961s trace: https://jaeger.dev.storj.io/trace/10ee146ebc418653 FAIL (2026-03-18 20:29:02 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001178628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X...X.........X....X....X.......X....................] duration: 6.24141684s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:29:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.241447154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX..X....X........X.....................................] duration: 6.145746503s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:28:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.145774977s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX..X....X........X....................................] duration: 5.564922662s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:28:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.564969577s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X...........] duration: 3.786173373s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:28:06 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.786201938s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 2.661195568s trace: https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594 FAIL (2026-03-18 20:28:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.661225784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X...X.........X....X....X.......X...................] duration: 1.705634415s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:27:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.70566013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX..X....X........X...................................] duration: 1.205572693s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:27:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.205601736s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 653.81115ms trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:27:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 653.844ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 549.714545ms trace: https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594 FAIL (2026-03-18 20:27:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 549.812772ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X...X.........X....X....X.......X..................] duration: 1.239246239s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:27:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.239284645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X..........] duration: 446.465201ms trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:27:10 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.495525ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X...X.........X....X....X.......X.................] duration: 1.275704695s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:26:52 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.275807097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XX..X....X........X..................................] duration: 1.306847536s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:26:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.306885508s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 553.551291ms trace: https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594 FAIL (2026-03-18 20:26:36 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.587389ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 765.024635ms trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:26:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 765.129916ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XX..X....X........X.................................] duration: 1.291666704s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:26:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.291703276s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..........X...X.........X....X....X.......X................] duration: 1.442679329s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:26:03 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.442777824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X..X.........] duration: 410.374998ms trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:25:57 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.400642ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 783.623331ms trace: https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594 FAIL (2026-03-18 20:25:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 783.680635ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XX..X....X........X................................] duration: 1.22489472s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:25:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.224979904s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)