Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 403.162942ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:46:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.192135ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 410.038178ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:46:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.083307ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 421.264634ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:45:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.375263ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 562.951192ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:44:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 563.037797ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 406.734495ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:44:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.00023ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 441.903206ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:43:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.931469ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 525.616197ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:43:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 525.736694ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 465.671186ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:42:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 465.700271ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 503.845246ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:42:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.879804ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 493.628326ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:41:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.66257ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 442.872745ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:40:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.912042ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 446.011164ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:39:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.049436ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 466.579758ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:38:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.606062ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 395.982499ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:37:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 396.023662ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 537.420644ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:36:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 537.451262ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 439.804702ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:35:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.835036ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 427.68979ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:34:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.716128ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.021153999s trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:34:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.021182889s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 4.431369775s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:13:12 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.431409582s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 4.498243466s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:12:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.498271571s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 4.406501055s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:11:11 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.406529083s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.453142604s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:10:48 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.453176821s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 4.393255535s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:09:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.393284284s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 4.279221905s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:09:11 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.279319069s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.245151461s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:08:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.24518036s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 4.364513663s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:08:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.364534525s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.376616383s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:07:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.376642354s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.35150104s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:06:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.351526616s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 4.421341258s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:06:10 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.42136904s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.434120099s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:05:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.434167226s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.393594815s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:05:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.393625959s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 4.374545432s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:04:25 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.374581175s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.206117336s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:03:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.206147314s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.18101291s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:02:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.181043796s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.406693237s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:01:37 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.406742147s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.547904825s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:01:06 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.547936674s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 4.601798265s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 17:00:30 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.601827802s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.580900428s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:59:37 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.580924581s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.147108055s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:58:28 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.147145254s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.168204916s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:58:06 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.168234514s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 4.316448279s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:57:41 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.316529583s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 4.297197089s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:56:58 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.297233756s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 4.230076453s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:56:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.230100035s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.307475974s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:56:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.307499808s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 4.270717985s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:55:11 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.270750725s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 4.334035441s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:54:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.334061804s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.298624805s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:53:31 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.298667183s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 4.859281216s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:52:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.85931261s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 4.16426549s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:51:47 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.164298293s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.365344251s trace: https://jaeger.dev.storj.io/trace/62d5ddb4d118eb FAIL (2025-12-14 16:50:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.365370037s core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)